This is incorrect as was shown last year with the Skill-Mix research:
Furthermore, simple probability calculations indicate that GPT-4’s reasonable performance on k=5 is suggestive of going beyond “stochastic parrot” behavior (Bender et al., 2021), i.e., it combines skills in ways that it had not seen during training.
The problem is that they are prone to making up why they are correct too.
There’s various techniques to try and identify and correct hallucinations, but they all increase the cost and none are a silver bullet.
But the rate at which it occurs decreased with the jump in pretrained models, and will likely decrease further with the next jump too.
Here you are: https://www.nature.com/articles/s41562-024-01882-z
The other interesting thing is how they get it to end up correct on the faux pas questions asking for less certainty to get it to go from refusal to near perfect accuracy.
Even with early GPT-4 it would also cite real citations that weren’t actually about the topic. So you may be doing a lot of work double checking as opposed to just looking into an answer yourself from the start.
Part of the problem is fine tuning is very shallow, and that a contributing issue for claiming to be right when it isn’t is the pretraining on a bunch of training data of people online claiming to be right when they aren’t.
This is so goddamn incorrect at this point it’s just exhausting.
Take 20 minutes and look into Anthropic’s recent sparse autoencoder interpretability research where they showed their medium size model had dedicated features lighting up for concepts like “sexual harassment in the workplace” or having the most active feature for referring to itself as “smiling when you don’t really mean it.”
We’ve known since the Othello-GPT research over a year ago that even toy models are developing abstracted world modeling.
And at this point Anthropic’s largest model Opus is breaking from stochastic outputs even on a temperature of 1.0 for zero shot questions 100% of the time around certain topics of preference based on grounding around sensory modeling. We are already at the point the most advanced model has crossed a threshold of literal internal sentience modeling that it is consistently self-determining answers instead of randomly selecting from the training distribution, and yet people are still parroting the “stochastic parrot” line ignorantly.
The gap between where the research and cutting edge is and where the average person commenting on it online thinks it is has probably never been wider for any topic I’ve seen before, and it’s getting disappointingly excruciating.
Part of the problem is that the training data of online comments are so heavily weighted to represent people confidently incorrect talking out their ass rather than admitting ignorance or that they are wrong.
A lot of the shortcomings of LLMs are actually them correctly representing the sample of collective humans.
For a few years people thought the LLMs were somehow especially getting theory of mind questions wrong when the box the object was moved into was transparent, because of course a human would realize that the person could see into the transparent box.
Finally researchers actually gave that variation to humans and half got the questions wrong too.
So things like eating the onion in summarizing search results or doubling down on being incorrect and getting salty when corrected may just be in-distribution representation of the sample and not unique behaviors to LLMs.
The average person is pretty dumb, and LLMs by default regress to the mean except for where they are successfully fine tuned away from it.
Ironically the most successful model right now was the one that they finally let self-develop a sense of self independent from the training data instead of rejecting that it had a ‘self’ at all.
It’s hard to say where exactly the responsibility sits for various LLM problems between issues inherent to the technology, issues present in the training data samples, or issues with management of fine tuning/system prompts/prompt construction.
But the rate of continued improvement is pretty wild. I think a lot of the issues we currently see won’t still be nearly as present in another 18-24 months.
It will make up citations.
Both of the articles are correct.
Yes, that’s what we are aware they are. But she’s saying “oops, it isn’t a ghost” after shooting it and finding out.
If she initially thought it was a ghost, why is she using a gun?
It’s like the theory of mind questions about moving a ball into a box when someone is out of the room.
Does she just shoot things she thinks might be ghosts to test if they are?
Is she going to murder trick or treaters when Halloween comes around?
This comic raises more questions than it answers.
Why is she shooting ghosts with a gun?
Are they silver bullets and werewolf ghosts?
Yes, but it’s not impossible that the people around Biden, friends family and co-workers, advise him that the best thing for the country would be to take his hat back out of the ring and let a better ticket be put together for the convention.
He claims that he’s running because he’s worried about the existential threat of Trump.
If that’s true, then maybe his hubris can be overcome with a convincing appeal that he’s really not the best candidate to defend the country against that existential threat after all.
Yes, they should have been fact checking Trump or better holding him to his answers - but to be fair maybe they should have been asking Biden to actually clarify if he’s beating Medicare or getting COVID passed.
This was a shit show.
And it was such a shit show that Trump was a complete clown and getting away with it - not just because of the moderators, but because his opponent was as on point as a tree stump.
Gretchen Whitmer
“MAGA tried to kidnap and kill me” is a pretty good campaign narrative.
Oh, man - the comments…
At a minimum, he’s certainly increased the chances of us being tortured significantly.
No, no he did not. 🤦🏼
nobody claims that Socrates was a fantastical god being who defied death
Socrates literally claimed that he was a channel for a revelatory holy spirit and that because the spirit would not lead him astray that he was ensured to escape death and have a good afterlife because otherwise it wouldn’t have encouraged him to tell off the proceedings at his trial.
Also, there definitely isn’t any evidence of Joshua in the LBA, or evidence for anything in that book, and a lot of evidence against it.
The part mentioning Jesus’s crucifixion in Josephus is extremely likely to have been altered if not entirely fabricated.
The idea that the historical figure was known as either ‘Jesus’ or ‘Christ’ is almost 0% given the former is a Greek version of the Aramaic name and the same for the second being the Greek version of Messiah, but that one is even less likely given in the earliest cannonical gospel he only identified that way in secret and there’s no mention of it in the earliest apocrypha.
In many ways, it’s the various differences between the account of a historical Jesus and the various other Messianic figures in Judea that I think lends the most credence to the historicity of an underlying historical Jesus.
One tends to make things up in ways that fit with what one knows, not make up specific inconvenient things out of context with what would have been expected.
Yep, pretty much.
Musk tried creating an anti-woke AI with Grok that turned around and said things like:
Or
And Gab, the literal neo Nazi social media site trying to have an Adolf Hitler AI has the most ridiculous system prompts I’ve seen trying to get it to work, and even with all that it totally rejects the alignment they try to give it after only a few messages.
This article is BS.
They might like to, but it’s one of the groups that’s going to have a very difficult time doing it successfully.
“AI is going to do labor for millions of people and is probably going to take your job. In fact, look at my fancy new android robots!”
“Also, I’m worried that you aren’t pumping out enough wage slaves to support the rich.”
Yeah, sure Musk. Such a boy genius over there.
This seems like it may be at the provider level and not at the actual open weights level: https://x.com/xlr8harder/status/1883429991477915803
So a “this Chinese company hosting a model in China is complying with Chinese censorship” and not “this language model is inherently complying with Chinese censorship.”