In my application, code generation, the distilled DeepSeek models (7B to 70B) perform poorly. They imitate the reasoning of the r1 model, but their conclusions are not correct.
The real r1 model is great, better than o1, but the distilled models are not even as good as the base models that they were distilled from.
The DeepSeek R1 paper explains how they trained their model in enough detail that people can replicate the process. Many people around the world are doing so, using various sizes of models and training data. Expect to see many posts like this over the next three months. The attempts that use small models will get done first. The larger models take much longer.
Small r1 style models are pretty limited, so this is interesting primarily from an “I reproduced the results” point of view, not a “here is a new model that’s useful” pov.
For distilled models, we apply only SFT and do not include an RL stage, even though incorporating RL could substantially boost model performance. Our primary goal here is to demonstrate the effectiveness of the distillation technique, leaving the exploration of the RL stage to the broader research community.
The impression I got from the paper, although I don't think it was explicitly stated, is that they think distillation will work better than training the smaller models using RL (as OP did).
> We demonstrate that the reasoning patterns of larger models can be distilled into smaller
models, resulting in better performance compared to the reasoning patterns discovered
through RL on small models
I found this statement from the paper to be at odds with what you cited, but I guess they mean SFT+RL would be better than either just SFT and RL
I think they're saying that some reasoning patterns which large models can learn using only RL (i.e. without the patterns existing in the training data), can't be learned by smaller models in the same way. They have to be 'taught' through examples provided during SFT.
You also want colleges to signal to their applicants, not force them to also signal for their alumni. The two will naturally be correlated, but you can do better by specializing.
“You should consider using this in your requirements” implies that this is not a hard rule, it’s just an ignorable suggestion. It would be interesting to audit gov.uk web pages over time to see whether this advice is being followed.
Don't forget the rules of British English that make it very clear that the grammatical construction: "you should consider" means "you must in all circumstances save for the immediate alternate outcome being a genocide."
SHOULD This word, or the adjective "RECOMMENDED", mean that there
may exist valid reasons in particular circumstances to ignore a
particular item, but the full implications must be understood and
carefully weighed before choosing a different course.
It means you can’t simply ignore it, and instead have to have compelling reasons to justify any deviation.
Unfortunately, in many organizations, "the library we use doesn't follow this recommendation" is a valid compelling reason. Which means that in practice "SHOULD" effectively means "WOULD BE NICE IF".
I remember seeing the PERQ at trade shows. The best thing about the PERQ was its monitor, which was unusually sharp for that era. It used a yellow-white long persistence phosphor. A CMU grad student friend told me that the monitor designer was “a close personal friend of the electron”, implying that the analog circuitry of the PERQ monitor was especially high quality.
They identify 6 bugs/mistakes, of which, not doing staged releases, was the final mistake.
They stop short of identifying the real root issues of running at kernel level, and of not auto-backing-out updates that cause crashes, perhaps because those causes are harder to fix.
A tool for updating bazel build target dependencies. It inspects build files and source code, then adds/removes dependencies from build targets as needed. It requires using global include paths in C/C++ sources. It is not perfect, but it is pretty nice!
That’s certainly how it seemed from the Android side at the time. The Linux side was hoping that they could adapt the desktop Linux stack to work well on mobile devices, without introducing new concepts like wakelocks. It took them a long time to give up on that approach.
https://opengoal.dev/ has reverse-engineered a compiler and runtime for the non-GC lisp GOAL that Naughty Dog used for the Jax and Daxter action adventure games. They have decompiled the game code for all three games.
My understanding is that Naughty Dog abandoned GOAL after the Jak games due to pressure from Sony. Sony wanted all of their studios to be able to share source code. [0]
I hate this. Any two organically grown codebases (like, for example, games from different studios) are going to be so different that significant, non-trivial code sharing between them is going to be impossible anyway. Anything sufficiently generic might as be distributed as a precompiled library, and then use the FFI in your language of choice to take advantage.
"Add this to the difficulty curve of learning a new language for new hires".
This argument is such a pointy-haired boss argument. Mature applications and systems will be more complicated and take longer to learn than the basics of pretty much any programming language. Grab some juniors and teach them if local seniors don't want to work in the language for a reasonable price.
Most seniors I know wouldn't balk at learning a new language for a job, because most seniors know what you just stated to be true. It will take much longer to get up to speed on the codebase than it will be to learn the language. Even for "difficult" languages like a lisp or Haskell.
Management and HR seem to assume it will take significantly longer to get up to speed in a new language, but don't seem to care that new hires have to learn all of their weird C++ idioms that have built up over decades like atherosclerosis.
Not to mention working programmers are expected to keep up with changes to C++, Python, Java, JavaScript (and its frameworks), Go2.0, etc., many of which constitute "new language" levels of different, not to mention actual language changes like JS -> TypeScript, or Java -> Kotlin, or ObjC -> Swift, and even occasionally mobile lang -> C++ (maybe just a shared core). There's plenty of evidence that it's not that bad. And meanwhile, Common Lisp hasn't changed, code from the 90s works unmodified, the only things to keep up with really are which libraries and implementation-specific features are new/interesting/in fashion (same as any language ecosystem).
It is quite different, as those are incremental changes, and most of them can be ignored until there is a requirement to use a library or SDK that makes more recent features a requirement.
Any corporate developer knows the pain of actually being allowed to upgrade toolchains, traditionally lagging behind several years behind lang v-latest.
Don't work for people that disallow upgrades and maintenance. Don't make deals with people who don't understand that software is finished when it's dismantled and the code deleted.
You can see them throughout Southern California. They are strikingly beautiful when in bloom. Not mentioned in the article is that the blossoms exude a sticky nectar that makes an absolute mess of the area under the tree.
The real r1 model is great, better than o1, but the distilled models are not even as good as the base models that they were distilled from.
reply