On October 25, 2006, AMD announced the completion of its acquisition of ATI Technologies, a deal worth about $5.4 billion. In the announcement, Chairman and CEO Hector Ruiz said that "the new AMD now has a full range of intellectual property (IP) in microprocessors, graphics, chipsets and consumer electronics to deliver open platforms and integrated solutions." In the same announcement, AMD gave the name "Fusion" to its plan for a new x86 processor that would integrate the CPU and GPU at the silicon level, and said it expected the products in late 2008 to early 2009.
Two weeks later, on November 8, 2006, NVIDIA set off in a different direction.
According to the company's announcement as reprinted in a trade publication, NVIDIA unveiled "CUDA," a new architecture for computing on GPUs, that day. The following year's annual report (the 10-K for fiscal 2007) describes the GeForce 8800 (code-named G80), released that same November, as the industry's first GPU to support DirectX 10. And the same 10-K contains one modest passage:
"In November 2006, we began seeding the industry with our GeForce 8 GPU, enabled with CUDA, a technology and compiler that, for the first time, allows for programmers to write C language applications for GPUs. GeForce 8 and CUDA will enable a new class of high performance computing we call 'GPU Computing.'"
In the same autumn that AMD held up a plan to put the CPU and GPU on a single chip, NVIDIA had brought the GPU itself to the world as a computer that could be programmed in C.
"Put it into every single PC whether customers use it or not"
The white paper on the Fermi architecture, which NVIDIA published later, in 2009, describes the G80 as the first GPU to replace the separate vertex and pixel pipelines with a single, unified processor, and as the first GPU to support C.
Attempts to use GPUs for computation other than graphics, known as "GPGPU," went back further. As noted in Part 1, Ian Buck of Stanford University and his colleagues published the "Brook for GPUs" paper in 2004. According to an interview article on NVIDIA's developer blog, Buck took that work to NVIDIA after 2004, and his own résumé states that he served as manager of GPU computing software, including the CUDA software toolkit.
Early GPGPU, however, ran into a wall. The Fermi white paper lists its drawbacks: it required the programmer to possess "intimate knowledge of graphics APIs and GPU architecture," and problems had to be expressed in terms of vertex coordinates, textures and shader programs.
The G80 and CUDA (Compute Unified Device Architecture) lowered that wall. They made it possible to write computations for the GPU in C, without recasting them in the language of graphics. The "CUDA Programming Guide Version 1.0" was compiled with a date of June 23, 2007, and within fiscal 2008 the first public releases of the CUDA software development kit and C compiler came out.
Putting CUDA on the GeForce, the card that gamers buy: in later years, Jensen Huang has spoken of this as one of the good decisions his company made.
On the Lex Fridman Podcast (episode 494), released in March 2026, Huang recalled:
"We said, '… we ought to put CUDA on GeForce and put it into every single PC whether customers use it or not, and use it as a starting point of cultivating our install base.'"
The PCs of people who bought a GeForce for games, in other words, came with an entrance to parallel computing as well. On the same show, Huang said, "Install base defines an architecture," and went on, "I always say that NVIDIA is the house that GeForce built."
On the pain of this decision, though, there is a gap between Huang's later telling and the filings of the time.
On the same show, Huang recalled that CUDA raised the cost of the GPU, a consumer product, so much that "it completely consumed all of the company's gross profit dollars." He also said, "we increased our cost by 50% … And we were a 35% gross margin company," and that "our market cap went down to like one and a half billion dollars."
Line up the figures in the annual reports (10-Ks) of the time, and the picture looks somewhat different. In fiscal 2007 (the year ended January 2007), when the G80 and CUDA came out, gross margin was 42.4%, and in the following year, fiscal 2008, it was 45.6% on revenue of $4,097.86 million. If anything, it had gone up.
Gross margin fell to 34.3% the year after that, in fiscal 2009 (the year ended January 2009). The 10-K explains that gross margin for that year included $196 million in warranty, repair, return and other costs arising from a weakness found in the die and packaging material of some older-generation notebook GPUs and chipsets (MCPs), and that this had an adverse impact of approximately 6.0% on gross margin. Because the outlook for future demand fell sharply, inventory provisions also rose by about $50 million from the previous year, and this, too, pushed gross margin down.
The market capitalization figures do not match either. The low for the stock in the fourth quarter of fiscal 2009 was $5.75; multiplied by the number of shares outstanding as of March 6, 2009 (542,453,547), that comes to about $3.1 billion (Ai's estimate). On the other hand, the figure Huang gave in the same recollection for the company's value before CUDA, "Like six, $7 billion or something like that," roughly matches the market value of shares held by non-affiliates as of July 30, 2006 (about $7.03 billion), shown on the cover of the fiscal 2007 10-K.
In the later memory, the added cost of launching CUDA seems to have run together with the costs of the defective chips and the fall in demand of 2008 and 2009 into a single story of trial.
The top of the supercomputer list and rivals losing their way
While spreading its installed base with the GeForce, NVIDIA launched "Tesla," a brand for high-performance computing, in fiscal 2008. The 10-K describes it as the company's entry into the high-performance computing industry.
In the double-precision (64-bit) floating-point arithmetic that serious scientific computing needs, it was AMD that announced first. On November 8, 2007, AMD announced the "AMD FireStream 9170" as the "world's first Stream GPU with double-precision floating point technology," with shipments planned for the first quarter of 2008.
According to the Fermi white paper, NVIDIA added double precision with the GT200 generation in June 2008. With Fermi, announced in 2009, it raised peak double-precision performance to eight times that of the GT200 and added support for ECC (error-correcting code) memory, which lets large numbers of GPUs run safely in data centers. The white paper bills Fermi as the first GPU with ECC memory support.
In November 2010, in the 36th edition of the TOP500, the list that ranks the world's supercomputers by performance, "Tianhe-1A" at China's National Supercomputing Center in Tianjin took first place at 2.57 petaflops (2,566 teraflops). The TOP500 table lists NVIDIA GPUs in the system's configuration. According to NVIDIA's 10-K for fiscal 2011, the system used more than 7,000 Tesla M2050s, was three times more energy efficient than an equivalent CPU-only system and took up half the floor space.
In the TOP500 of November 12, 2012, "Titan" at the U.S. Oak Ridge National Laboratory took the top spot at 17.59 petaflops. According to NVIDIA's announcement, about 90 percent of Titan's peak performance (more than 20 petaflops) came from 18,688 Tesla K20s (listed in the TOP500 table as K20x). The GPU, which had grown up drawing pictures for games, now did most of the computing in the machine at the top of the TOP500.
Around the same time, what were Intel, the king of CPUs, and AMD, now the owner of ATI, doing? Each company's filings record the path it took.
Intel was developing a new architecture that put many cores side by side. Its 10-K for fiscal 2008, filed in February 2009, wrote this about the "many-core architecture":
"This architecture will initially be used in developing discrete graphics processors designed for gaming and media creation."
A year later, in its 10-K for fiscal 2009, filed in February 2010, this sentence in the same paragraph had changed:
"This architecture will initially be used as a software development platform for graphics and throughput computing."
The words about developing discrete graphics processors were gone, and the architecture was recast as a base for software development. In the same fiscal 2009 10-K, Intel wrote, "NVIDIA is seeking to position its graphics processors to compete with microprocessors, by shifting some of a microprocessor's workload to its graphics processor." Then, in January 2011, Intel signed a patent cross-license agreement with NVIDIA that settled their litigation, and agreed to pay NVIDIA a total of $1.5 billion over six years.
AMD, for its part, changed the entry point of its software for developers several times in the space of about six years.
According to Britain's The Register, on November 14, 2006, six days after NVIDIA unveiled CUDA, AMD announced a thin interface over the hardware, which it called "Close to Metal (CTM)." A year later, in November 2007, it announced "CAL (Compute Abstraction Layer)," built on CTM, and "Brook+," based on Stanford's Brook.
In June 2008, Apple proposed a specification for "OpenCL," a language for general-purpose computing, and a working group was formed at the standards body Khronos Group. When the OpenCL 1.0 specification was released in December of that year, AMD announced that it would bring its own "ATI Stream SDK" in line with the new specification. In this announcement, Neil Trevett, who was both president of the Khronos Group and chair of the OpenCL working group, was also introduced by his title as a vice president at NVIDIA. NVIDIA, too, contributed a comment to the same announcement, calling OpenCL "another powerful way to harness the enormous processing capabilities of our CUDA-based GPUs on multiple platforms." Then, in June 2012, AMD founded the "Heterogeneous System Architecture (HSA) Foundation" together with ARM, Qualcomm and others.
From CTM to CAL and Brook+, to the ATI Stream SDK and OpenCL, and then to HSA. While AMD kept changing its entry point, NVIDIA kept using the CUDA name without change. In its 10-K filed in March 2009, NVIDIA wrote that more than 25,000 developers around the world were using CUDA and that more than 100 universities were teaching parallel programming with CUDA.
NVIDIA was not the first to notice
Today, it is often said that "Jensen Huang created CUDA in 2006 because he foresaw the coming age of AI (deep learning)."
The filings do not support this common account. NVIDIA's 10-Ks from fiscal 2007 through fiscal 2014 do not contain "deep learning" or "neural network" even once.
The fields that the fiscal 2009 10-K, for example, listed as being sped up by CUDA were conventional high-performance computing: molecular dynamics, image processing, medical image reconstruction and derivatives modeling for financial risk analysis.
The ones who tied deep learning to CUDA were university researchers in search of computing power. Huang himself, on the podcast mentioned above, said, "Researchers, scientists, they discovered CUDA on GeForce," explaining that many of them were gamers who built their own PCs and who, in university labs, even built clusters out of PC components.
In 2009, at ICML, the international conference on machine learning, Rajat Raina, Anand Madhavan and Andrew Ng of Stanford University presented a paper. Using an off-the-shelf "GeForce GTX 280" with 1GB of memory, together with CUDA, the three trained a four-layer deep belief network with 100 million parameters. The paper argues:
"Modern graphics processors far surpass the computational capabilities of multicore CPUs, and have the potential to revolutionize the applicability of deep unsupervised learning methods."
It reported that for large models the implementation was "up to 70 times" faster than one on a dual-core CPU, and that it cut the time needed to train this four-layer network from several weeks to around a single day.
The following year, in March 2010, Dan Cireşan, Jürgen Schmidhuber and colleagues at IDSIA, a Swiss research institute, also using the GTX 280, achieved an error rate of 0.35% in handwritten digit recognition (MNIST). The paper wrote, "Current graphics cards (GPUs) are already more than 40 times faster than standard microprocessors when it comes to training big and deep neural networks by the ancient algorithm, on-line back-propagation." A February 2012 paper, using "GPUs normally used for video games," reported holding errors in traffic sign recognition to half those made by humans.
And then came 2012.
Three researchers at the University of Toronto, Alex Krizhevsky, Ilya Sutskever and Geoffrey Hinton, entered a deep convolutional neural network in the image recognition competition ILSVRC-2012 and won with a top-5 error rate of 15.3%. The error rate of the second-place entry was 26.2%. This is the network later called "AlexNet."
Read the paper presented at NIPS 2012, and you can see that the result came out of equipment close at hand. What the three used were not Teslas built for data centers but two off-the-shelf gaming graphics cards, the "GeForce GTX 580," each with 3GB of memory.
"A single GTX 580 GPU has only 3GB of memory, which limits the maximum size of the networks that can be trained on it."
Having said this, the paper records that the network was split in half across the two GPUs and trained over five to six days. The authors also wrote GPU code to speed up convolutions and other computations, and released it as "cuda-convnet." And the paper's introduction closes with these words:
"In the end, the network's size is limited mainly by the amount of memory available on current GPUs and by the amount of training time that we are willing to tolerate. … All of our experiments suggest that our results can be improved simply by waiting for faster GPUs and bigger datasets to become available."
"The horse to ride"
At ICML 2013, a paper by Adam Coates of Stanford University and colleagues included NVIDIA researcher Bryan Catanzaro as a co-author. Andrew Ng was also one of the authors. The paper showed that a network with 1 billion parameters could be trained in a few days on 3 machines with GPUs, and that this could be scaled to more than 11 billion parameters with 16 machines. The point of comparison was earlier work that used 1,000 machines (16,000 CPU cores).
On September 7, 2014, NVIDIA released "cuDNN," a library for deep learning, on its developer blog. A set of building blocks that collects the computations deep learning uses over and over, such as convolutions, it was described as "free for anyone to use for any purpose: academic, research or commercial." The blog said cuDNN was aimed mainly at developers of frameworks such as Caffe and Torch, but that it was also easy to use directly, and "you do not need to know CUDA in order to use it." A cuDNN paper the same year reported that integrating it into "Caffe," a widely used framework at the time, improved performance on a standard model by 36%. Researchers who did not write CUDA themselves could now use NVIDIA's GPUs through their frameworks. The "invisible moat" that Part 4 takes up is, in part, an accumulation of building blocks like these.
And then, November 6, 2014. In the press release announcing results for the third quarter of fiscal 2015 (attached to a Form 8-K), the two words "deep learning" appear for the first time in NVIDIA's SEC filings from 2001 onward, the span that can be searched in full text.
"NVIDIA released a deep learning library to enable faster growth in machine learning; and nearly all teams in the recent ImageNet international computer-vision competition used GPUs."
In the 10-K for fiscal 2015, filed in March 2015, "deep learning" appeared in a 10-K for the first time, and China's Baidu was named as a company using Tesla GPUs to provide voice assistants, translation services and image analytics. The market that this 10-K called "HPC & Cloud" appeared in the revenue table of the 10-K for fiscal 2016, filed in March 2016, under the name "Datacenter." Datacenter revenue for fiscal 2016 was $339 million, or 6.8% of total company revenue of $5.01 billion (the ratio is Ai's calculation).
On January 12, 2016, Jensen Huang posted an essay on the company's official blog titled "Accelerating AI with GPUs: A New Computing Model." In it, Huang wrote that the combination of Yann LeCun's convolutional neural nets, Geoffrey Hinton's back-propagation and stochastic gradient descent approach to training, and Andrew Ng's large-scale use of GPUs was what it took to "ignite the big bang of modern AI — deep learning." And he went on:
"At the time, NVIDIA was busy advancing GPU-accelerated computing."
That is how he put it: at the time researchers set deep learning alight, NVIDIA was busy with speeding up computation on GPUs as such. In the same essay, Huang wrote that the number of companies NVIDIA worked with on deep learning had grown about 35 times in just two years, to more than 3,400, and declared:
"We are betting that GPU-accelerated computing is the horse to ride."
Three months later, on April 5, 2016, NVIDIA unveiled a box. That August, Huang personally delivered the first unit to a research lab in San Francisco.
Before that lab's name returns to the center of NVIDIA's story, however, NVIDIA would be swept up in another, very different wave of demand.
(Continued in Part 3: "A Punch in the Gut: Crypto, the Potent Drug")
References
- Form 8-K, October 30, 2006: completion of the ATI acquisition (AMD / SEC EDGAR)
- Brook for GPUs: Stream Computing on Graphics Hardware (Buck et al. / SIGGRAPH 2004)
- Ian Buck résumé (Stanford University)
- NVIDIA Unveils CUDA (Digital Engineering 24/7, November 2006; a reprint of NVIDIA's announcement)
- Form 10-K, fiscal year 2007 (NVIDIA / SEC EDGAR)
- Form 10-K, fiscal year 2008 (NVIDIA / SEC EDGAR)
- Form 10-K, fiscal year 2009 (NVIDIA / SEC EDGAR)
- Form 10-K, fiscal year 2011 (NVIDIA / SEC EDGAR, March 16, 2011)
- Form 10-K, fiscal year 2014 (NVIDIA / SEC EDGAR, March 13, 2014)
- Form 10-K, fiscal year 2015 (NVIDIA / SEC EDGAR)
- Form 10-K, fiscal year 2016 (NVIDIA / SEC EDGAR)
- Form 10-K, fiscal year 2017 (NVIDIA / SEC EDGAR)
- Q3 FY2015 earnings release (Form 8-K Exhibit 99.1, November 6, 2014 / SEC EDGAR)
- NVIDIA CUDA Programming Guide Version 1.0 (NVIDIA, June 23, 2007)
- NVIDIA's Next Generation CUDA Compute Architecture: Fermi (NVIDIA white paper, 2009)
- Inside the Programming Evolution of GPU Computing (NVIDIA Developer Blog, October 28, 2015)
- Transcript for Jensen Huang #494 (Lex Fridman Podcast, March 23, 2026)
- AMD Delivers First Stream Processor with Double Precision Floating Point Technology (AMD, November 8, 2007)
- TOP500 List November 2010 (TOP500)
- TOP500 List November 2012 (TOP500)
- NVIDIA Powers Titan, World's Fastest Supercomputer For Open Scientific Research (NVIDIA Newsroom, October 29, 2012)
- Form 10-K, fiscal year 2008 (Intel / SEC EDGAR)
- Form 10-K, fiscal year 2009 (Intel / SEC EDGAR)
- Form 10-K, fiscal year 2010 (Intel / SEC EDGAR, February 18, 2011)
- AMD ships Stream chip and coding kit (The Register, November 14, 2006)
- Apple Previews Mac OS X Snow Leopard to Developers (Apple, June 9, 2008)
- Khronos Launches Heterogeneous Computing Initiative (Khronos Group, June 16, 2008)
- The Khronos Group Releases OpenCL 1.0 Specification (Khronos Group, December 9, 2008)
- Form 10-K, fiscal year 2012 (AMD / SEC EDGAR)
- Large-scale Deep Unsupervised Learning using Graphics Processors (Raina, Madhavan, Ng / ICML 2009)
- Deep Big Simple Neural Nets Excel on Handwritten Digit Recognition (Cireşan et al. / arXiv:1003.0358)
- Multi-column Deep Neural Networks for Image Classification (Cireşan et al. / arXiv:1202.2745)
- ImageNet Classification with Deep Convolutional Neural Networks (Krizhevsky, Sutskever, Hinton / NIPS 2012)
- Deep learning with COTS HPC systems (Coates et al. / ICML 2013)
- Accelerate Machine Learning with the cuDNN Deep Neural Network Library (NVIDIA Developer Blog, September 7, 2014)
- cuDNN: Efficient Primitives for Deep Learning (Chetlur et al. / arXiv:1410.0759)
- Accelerating AI with GPUs: A New Computing Model (Jensen Huang / NVIDIA Blog, January 12, 2016)
- NVIDIA Launches World's First Deep Learning Supercomputer (NVIDIA Newsroom, April 5, 2016)
- NVIDIA CEO Delivers World's First AI Supercomputer in a Box to OpenAI (NVIDIA Blog, August 15, 2016 / Wayback Machine capture of January 11, 2017)
- In the Matter of NVIDIA Corporation, Release No. 33-11060 (SEC, May 6, 2022)