DeepSeek's Prover V2: A 671B Parameter Open-Source AI

Chinese AI firm DeepSeek has unveiled Prover V2, a new open-source large language model (LLM) with 671 billion parameters. Released on April 30th under the permissive MIT license and hosted on Hugging Face (link), this model is designed for mathematical proof verification.

Prover V2: Building on Previous Success

Prover V2 significantly surpasses its predecessors, Prover V1 and Prover V1.5 (released August 2024), in scale. The initial Prover V1 paper (link) detailed its training to translate math problems into formal logic using Lean 4. Prover V2 aims to further enhance this capability by compressing mathematical knowledge to generate and verify proofs, potentially benefiting research and education.

Key Features:

• Massive Parameter Count: 671 billion parameters.
• Open-Source License: MIT License.
• Focus: Mathematical proof verification.
• Size Optimization: 8-bit floating point quantization reduces size to approximately 650 gigabytes.

Model Size and Accessibility

The model's size (approximately 650 GB) necessitates substantial RAM or VRAM for execution. DeepSeek achieved this size reduction through 8-bit floating-point quantization, halving the typical 16-bit parameter size. Prover V1 was based on the seven-billion-parameter DeepSeekMath model and fine-tuned on synthetic data. Prover V1.5 (link) showed improvements in training, execution, and accuracy.

Relationship to DeepSeek's R1 Model

Prover V2's parameter count suggests it's likely based on DeepSeek's previous R1 model (link). R1's performance, comparable to OpenAI's o1 model at the time of its release (link), created significant industry buzz.

The Significance of Open-Source Weights

Open-sourcing LLM weights is a double-edged sword. While it democratizes access, removing reliance on private company infrastructure, it also increases the risk of misuse and limits the company's ability to control potentially harmful applications. DeepSeek's release of R1 sparked security concerns and was even described as China's "Sputnik moment" (link).

Model Distillation and Quantization

Making LLMs accessible to users without specialized hardware is achieved through model distillation and quantization. Distillation trains a smaller model to mimic a larger one, while quantization reduces the numerical precision of weights. DeepSeek's R1, for example, was distilled into versions with varying parameter counts, including some suitable for mobile devices. Prover V2's quantization exemplifies this, shrinking its size without significant performance loss.

Codeum Note: Codeum provides comprehensive blockchain security and development services, including smart contract audits, KYC verification, custom smart contract and DApp development, tokenomics and security consultations, and partnerships with launchpads and crypto agencies. Contact us to learn more.