Google data centers explained

Google data centers are the large data center facilities Google uses to provide their services, which combine large drives, computer nodes organized in aisles of racks, internal and external networking, environmental controls (mainly cooling and humidification control), and operations software (especially as concerns load balancing and fault tolerance).

There is no official data on how many servers are in Google data centers, but Gartner estimated in a July 2016 report that Google at the time had 2.5 million servers. This number is changing as the company expands capacity and refreshes its hardware.^[1]

Locations

The locations of Google's various data centers by continent are as follows:^{[2] [3]}

!Continent!Location!Geo!Products Location!Cloud Location!Timeline!Description

North America	Arcola (VA), USA	38.9433°N -77.5243°W	Loudoun County	N. Virginia (us-east4)	2017 - announced^[4] ^[5]
North America	Atlanta (GA), USA	33.7497°N -84.5848°W	Douglas County	-	2003 - launched	350 employees
South America	Cerrillos, Santiago, Chile	^[6]	-	Santiago (southamerica-west1)	2020 - announced^[7] 2021 - launched^[8]
Asia	Changhua County, Taiwan	24.1385°N 120.4257°W	Changhua County	Taiwan (asia-east1)	2011 - announced2013 - launched	60 employees
North America	Clarksville (TN), USA		Montgomery County	-	2015 - announced
North America	Columbus (OH), USA		-	Columbus (us-east5)	2022 - launched^[9]
North America	Council Bluffs (IA), USA		Council Bluffs		2007 - announced 2009 - completed first phase completed 2012 and 2015 - expanded	130 employees
North America	Council Bluffs (IA), USA			Iowa (us-central1)
Asia	Delhi, India		-	Delhi (asia-south2)	2020 - announced2021 - launched^[10]
Middle East	Doha, Qatar		-	Doha (me-central1)	2023 - launched^[11]
Europe	Dublin, Ireland	53.3201°N -6.4421°W	Dublin	-	2011 - announced2012 - launched	150 employees^[12]
Europe	Eemshaven, Netherlands		Eemshaven		2014 - announced2016 - launched 2018, 2019 - expansion	200 employees
Europe	Frankfurt, Germany	^[13]	-	Frankfurt (europe-west3)	2022 - expanded^[14]
Europe	Fredericia, Denmark	55.5582°N 9.6558°W	Fredericia	-	2018 - announced^[15] 2020 - launched	€600M building costs
Europe	Ghlin, Hainaut, Belgium	50.4693°N 3.8655°W	Saint-Ghislain	Belgium (europe-west1)	2007 - announced2010 - launched	12 employees
Europe	Hamina, Finland	60.5366°N 27.117°W	Hamina	Finland (europe-north1)	2009 - announced2011 - first phase completed 2022 - expansion	6 buildings, 400 employees ^[16]
North America	Henderson (NV), USA		Henderson	Las Vegas (us-west4)	2019 - announced^[17] 2020 - launched	64-acres; $1.2B building costs^[18] ^[19]
Asia	Hong Kong, Hong Kong		-	Hong Kong (asia-east2)	2017 - announced^[20] 2018 - launched^[21]
Asia	Inzai, Japan		Inzai	-	2023 - launched
Asia	Jakarta, Indonesia		-	Jakarta (asia-southeast2)	2020 - launched^[22]
Asia	Koto-Ku, Tokyo, Japan		-	Tokyo (asia-northeast1)	2016 - launched^[23]
North America	Leesburg (VA), USA	39.0518°N -77.539°W	Loudoun County	N. Virginia (us-east4)	2017 - announced
North America	Lenoir (NC), USA	35.8986°N -81.5474°W	Lenoir	-	2007 - announced 2009 - launched	over 110 employees
Asia	Lok Yang Way, Pioneer, Singapore	^[24]	Singapore	Singapore (asia-southeast1)	2022 - launched
Europe	London, UK		-	London (europe-west2)	2017 - launched^[25]
North America	Los Angeles (CA), USA		-	Los Angeles (us-west2)
Europe	Madrid, Spain		-	Madrid (europe-southwest1)	2022 - launched^[26]
Pacific	Melbourne, Australia		-	Melbourne (australia-southeast2)	2021 - launched^[27]
Europe	Middenmeer, Noord-Holland, The Netherlands	^[28]	Middenmeer	Netherlands (europe-west4)	2019 - announced^[29]
North America	Midlothian (TX), USA		Midlothian	Dallas (us-south1)	2019 - announced2022 - launched^[30]	375-acres; $600M building costs^[31]
Europe	Milan, Italy		-	Milan (europe-west8)	2022 - launched^[32]
North America	Moncks Corner (SC), USA		Berkeley County	South Carolina (us-east1)	2007 - launched 2013 - expanded	150 employees
North America	Montreal, Quebec, Canada^[33]		-	Montréal (northamerica-northeast1)	2018 - launched^[34]	62.4-hectares; $600M building costs^[35]
Asia	Mumbai, India		-	Mumbai (asia-south1)	2017 - launched^[36]
North America	New Albany (OH), USA		New Albany	-	2019 - announced	400-acres; $600M building costs^[37] ^[38]
Asia	Osaka, Japan		-	Osaka (asia-northeast2)	2019 - launched^[39]
South America	Osasco, São Paulo, Brazil		-	São Paulo (southamerica-east1)	2017 - launched^[40]
North America	Papillion (NE), USA		Papillion	-	2019 - announced	275-acres; $600M building costs^[41] ^[42]
Europe	Paris, France		-	Paris (europe-west9)	2022 - launched^[43]
North America	Pryor Creek (OK), USA	36.2411°N -95.3301°W	Mayes County	-	2007 - announced 2012 - expanded
South America	Quilicura, Santiago, Chile	-33.3585°N -70.6973°W	Quilicura	-	2012 - announced2015 - launched	up to 20 employees expected. A million dollar investment plan to increase capacity at Quilicura was announced in 2018.^[45]
North America	Reno (NV), USA		Storey County	-	2017 - 1,210 acres of land bought in the Tahoe Reno Industrial Center^[46] 2018 - announced 2018 November - project approved by the state of Nevada^[47] ^[48]
North America	Salt Lake City (UT), USA		-	Salt Lake City (us-west3)	2020 - launched^[49]
Asia	Seoul, South Korea		-	Seoul (asia-northeast3)	2020 - launched^[50]
Pacific	Sydney, Australia		-	Sydney (australia-southeast1)	2017 - launched^[51]
Middle East	Tel Aviv, Israel^[52]		-	Tel Aviv (me-west1)	2022 - launched^[53]
North America	The Dalles (OR), USA	45.6325°N -121.2023°W	The Dalles	Oregon (us-west1)	2006 - launched	80 full-time employees
North America	Toronto, Canada		-	Toronto (northamerica-northeast2)	2021 - launched^[54]
Europe	Turin, Italy		-	Turin (europe-west12)	2023 - launched^[55]
South America	Vinhedo, São Paulo, Brazil			São Paulo (southamerica-east1)
Europe	Warsaw, Poland		-	Warsaw (europe-central2)	2019 - announced2021 - launched^[56]
Asia	Wenya, Jurong West, Singapore	1.3513°N 103.7098°W	Singapore	Singapore (asia-southeast1)	2011 - announced2013 - launched 2015 - expanded
North America	Widows Creek (Bridgeport) (AL), USA	34.9134°N -85.7481°W^[57]	Jackson County	-	2018 - broke ground
Europe	Zürich, Switzerland	^[58]	-	Zurich (europe-west6)	2018 - announced2019 - launched^[59]	url=https://www.datacenterknowledge.com/google-alphabet/google-building-cloud-data-centers-close-swiss-banks	title=Google Building Cloud Data Centers Close to Swiss Banks
Europe	Austria				2022 - announced^[60]
Europe	Berlin, Germany^[61]			Berlin (europe-west10)	2021 - announced^[62] 2023 August - launched ^[63]
Middle East	Dammam, Saudi Arabia				2021 - announced^[64]
Europe	Athens, Greece				2022 - announced
North America	Kansas City, Missouri				2019 - announced^[65]
Middle East	Kuwait				2023 - announced^[66]
Asia	Malaysia				2022 - announced^[67]
Pacific	Auckland, New Zealand				2022 - announced
Europe	Oslo, Norway				2022 - announced
North America	Querétaro, Mexico				2022 - announced^[68]
Africa	Johannesburg, South Africa			Johannesburg (africa-south1)	2022 - announced2024 - launched
Europe	Sweden				2022 - announced
Asia	Tainan City, Taiwan		-	Taiwan (asia-east1)	2019 September - announced^[69] ^[70] ^[71]
Asia	Thailand				2022 - announced
Asia	Yunlin County, Taiwan		-	Taiwan (asia-east1)	2020 September - announced^[72]
Europe	Waltham Cross, Hertfordshire, UK				2024 January - announced^[73]

Hardware

Original hardware

The original hardware (circa 1998) that was used by Google when it was located at Stanford University included:^[74]

• Sun Microsystems Ultra II with dual 200 MHz processors, and 256 MB of RAM. This was the main machine for the original Backrub system.
• 2 × 300 MHz dual Pentium II servers donated by Intel, they included 512 MB of RAM and 10 × 9 GB hard drives between the two. It was on these that the main search ran.
• F50 IBM RS/6000 donated by IBM, included 4 processors, 512 MB of memory and 8 × 9 GB hard disk drives.
• Two additional boxes included 3 × 9 GB hard drives and 6 x 4 GB hard disk drives respectively (the original storage for Backrub). These were attached to the Sun Ultra II.
• SSD disk expansion box with another 8 × 9 GB hard disk drives donated by IBM.
• Homemade disk box which contained 10 × 9 GB SCSI hard disk drives.

Production hardware

As of 2014, Google has used a heavily customized version of Debian Linux. They migrated from a Red Hat-based system incrementally in 2013.^[75]

The customization goal is to purchase CPU generations that offer the best performance per dollar, not absolute performance. How this is measured is unclear, but it is likely to incorporate running costs of the entire server, and CPU power consumption could be a significant factor.^[76] Servers as of 2009–2010 consisted of custom-made open-top systems containing two processors (each with several cores), a considerable amount of RAM spread over 8 DIMM slots housing double-height DIMMs, and at least two SATA hard disk drives connected through a non-standard ATX-sized power supply unit. The servers were open top so more servers could fit into a rack. According to CNET and a book by John Hennessy, each server had a novel 12-volt battery to reduce costs and improve power efficiency.^{[77] [78]}

According to Google, their global data center operation electrical power ranges between 500 and 681 megawatts.^{[79] [80]} The combined processing power of these servers might have reached from 20 to 100 petaflops in 2008.^[81]

Network topology

Details of the Google worldwide private networks are not publicly available, but Google publications make references to the "Atlas Top 10" report that ranks Google as the third largest ISP behind Level 3.

In order to run such a large network, with direct connections to as many ISPs as possible at the lowest possible cost, Google has a very open peering policy.

From this site, we can see that the Google network can be accessed from 67 public exchange points and 69 different locations across the world. As of May 2012, Google had 882 Gbit/s of public connectivity (not counting private peering agreements that Google has with the largest ISPs). This public network is used to distribute content to Google users as well as to crawl the internet to build its search indexes.The private side of the network is a secret, but a recent disclosure from Google indicate that they use custom built high-radix switch-routers (with a capacity of 128 × 10 Gigabit Ethernet port) for the wide area network. Running no less than two routers per datacenter (for redundancy) we can conclude that the Google network scales in the terabit per second range (with two fully loaded routers the bi-sectional bandwidth amount to 1,280 Gbit/s).

These custom switch-routers are connected to DWDM devices to interconnect data centers and point of presences (PoP) via dark fiber.

From a datacenter view, the network starts at the rack level, where 19-inch racks are custom-made and contain 40 to 80 servers (20 to 40 1U servers on either side, while new servers are 2U rackmount systems.^[82] Each rack has an Ethernet switch). Servers are connected via a 1 Gbit/s Ethernet link to the top of rack switch (TOR). TOR switches are then connected to a gigabit cluster switch using multiple gigabit or ten gigabit uplinks.^[83] The cluster switches themselves are interconnected and form the datacenter interconnect fabric (most likely using a dragonfly design rather than a classic butterfly or flattened butterfly layout^[84]).

From an operation standpoint, when a client computer attempts to connect to Google, several DNS servers resolve www.google.com into multiple IP addresses via Round Robin policy. Furthermore, this acts as the first level of load balancing and directs the client to different Google clusters. A Google cluster has thousands of servers, and once the client has connected to the server additional load balancing is done to send the queries to the least loaded web server. This makes Google one of the largest and most complex content delivery networks.^[85]

Google has numerous data centers scattered around the world. At least 12 significant Google data center installations are located in the United States. The largest known centers are located in The Dalles, Oregon; Atlanta, Georgia; Reston, Virginia; Lenoir, North Carolina; and Moncks Corner, South Carolina.^[86] In Europe, the largest known centers are in Eemshaven and Groningen in the Netherlands and Mons, Belgium. Google's Oceania Data Center is located in Sydney, Australia.^[87]

Data center network topology

To support fault tolerance, increase the scale of data centers and accommodate low-radix switches, Google has adopted various modified Clos topologies in the past.^[88]

Project 02

One of the largest Google data centers is located in the town of The Dalles, Oregon, on the Columbia River, approximately 80 miles (129 km) from Portland. Codenamed "Project 02", the complex was built in 2006 and is approximately the size of two American football fields, with cooling towers four stories high.^{[89] [90]} The site was chosen to take advantage of inexpensive hydroelectric power, and to tap into the region's large surplus of fiber optic cable, a remnant of the dot-com boom. A blueprint of the site appeared in 2008.^[91]

Summa papermill

In February 2009, Stora Enso announced that they had sold the Summa paper mill in Hamina, Finland to Google for 40 million Euros.^{[92] [93]} Google invested 200 million euros on the site to build a data center and announced additional 150 million euro investment in 2012.^{[94] [95]} Google chose this location due to the availability and proximity of renewable energy sources.^[96]

Floating data centers

See also: Google barges. In 2013, the press revealed the existence of Google's floating data centers along the coasts of the states of California (Treasure Island's Building 3) and Maine. The development project was maintained under tight secrecy. The data centers are 250 feet long, 72 feet wide, 16 feet deep. The patent for an in-ocean data center cooling technology was bought by Google in 2009^{[97] [98]} (along with a wave-powered ship-based data center patent in 2008^{[99] [100]}). Shortly thereafter, Google declared that the two massive and secretly-built infrastructures were merely "interactive learning centers, [...] a space where people can learn about new technology."^[101]

Google halted work on the barges in late 2013 and began selling off the barges in 2014.^{[102] [103]}

Software

Most of the software stack that Google uses on their servers was developed in-house.^[104] According to a well-known Google employee, C++, Java, Python and (more recently) Go are favored over other programming languages.^[105] For example, the back end of Gmail is written in Java and the back end of Google Search is written in C++.^[106] Google has acknowledged that Python has played an important role from the beginning, and that it continues to do so as the system grows and evolves.^[107]

The software that runs the Google infrastructure includes:^[108]

• Google Web Server (GWS) custom Linux-based Web server that Google uses for its online services.
• Storage systems:
  • Google File System and its successor, Colossus^[109]
  • Bigtable structured storage built upon GFS/Colossus
  • Spanner planet-scale database, supporting externally-consistent distributed transactions
  • Google F1 a distributed, quasi-SQL DBMS based on Spanner, substituting a custom version of MySQL.
• Chubby lock service
• MapReduce and Sawzall programming language
• Indexing/search systems:
  • TeraGoogle Google's large search index (launched in early 2006)^[110]
  • Caffeine (Percolator) continuous indexing system (launched in 2010).
  • Hummingbird major search index update, including complex search and voice search.^[111]
• Borg declarative process scheduling software

Google has developed several abstractions which it uses for storing most of its data:^[112]

• Protocol Buffers "Google's lingua franca for data",^[113] a binary serialization format which is widely used within the company.
• SSTable (Sorted Strings Table) a persistent, ordered, immutable map from keys to values, where both keys and values are arbitrary byte strings. It is also used as one of the building blocks of Bigtable.^[114]
• RecordIO a sequence of variable sized records.^{[115] [116]}

Software development practices

Most operations are read-only. When an update is required, queries are redirected to other servers, so as to simplify consistency issues. Queries are divided into sub-queries, where those sub-queries may be sent to different ducts in parallel, thus reducing the latency time.

To lessen the effects of unavoidable hardware failure, software is designed to be fault tolerant. Thus, when a system goes down, data is still available on other servers, which increases reliability.

Search infrastructure

Index

Like most search engines, Google indexes documents by building a data structure known as inverted index. Such an index obtains a list of documents by a query word. The index is very large due to the number of documents stored in the servers.

The index is partitioned by document IDs into many pieces called shards. Each shard is replicated onto multiple servers. Initially, the index was being served from hard disk drives, as is done in traditional information retrieval (IR) systems. Google dealt with the increasing query volume by increasing number of replicas of each shard and thus increasing number of servers. Soon they found that they had enough servers to keep a copy of the whole index in main memory (although with low replication or no replication at all), and in early 2001 Google switched to an in-memory index system. This switch "radically changed many design parameters" of their search system, and allowed for a significant increase in throughput and a large decrease in latency of queries.^[117]

In June 2010, Google rolled out a next-generation indexing and serving system called "Caffeine" which can continuously crawl and update the search index. Previously, Google updated its search index in batches using a series of MapReduce jobs. The index was separated into several layers, some of which were updated faster than the others, and the main layer wouldn't be updated for as long as two weeks. With Caffeine, the entire index is updated incrementally on a continuous basis. Later Google revealed a distributed data processing system called "Percolator"^[118] which is said to be the basis of Caffeine indexing system.^{[119] [120]}

Server types

Google's server infrastructure is divided into several types, each assigned to a different purpose:^{[121] [122] [123]}

• Web servers coordinate the execution of queries sent by users, then format the result into an HTML page. The execution consists of sending queries to index servers, merging the results, computing their rank, retrieving a summary for each hit (using the document server), asking for suggestions from the spelling servers, and finally getting a list of advertisements from the ad server.
• Data-gathering servers are permanently dedicated to spidering the Web. Google's web crawler is known as GoogleBot. They update the index and document databases and apply Google's algorithms to assign ranks to pages.
• Each index server contains a set of index shards. They return a list of document IDs ("docid"), such that documents corresponding to a certain docid contain the query word. These servers need less disk space, but suffer the greatest CPU workload.
• Document servers store documents. Each document is stored on dozens of document servers. When performing a search, a document server returns a summary for the document based on query words. They can also fetch the complete document when asked. These servers need more disk space.
• Ad servers manage advertisements offered by services like AdWords and AdSense.
• Spelling servers make suggestions about the spelling of queries.

Security

In October 2013, The Washington Post reported that the U.S. National Security Agency intercepted communications between Google's data centers, as part of a program named MUSCULAR.^{[124] [125]} This wiretapping was made possible because, at the time, Google did not encrypt data passed inside its own network.^[126] This was rectified when Google began encrypting data sent between data centers in 2013.^[127]

Environmental impact

Google's most efficient data center runs at using only fresh air cooling, requiring no electrically powered air conditioning.^[128]

In December 2016, Google announced that—starting in 2017—it would purchase enough renewable energy to match 100% of the energy usage of its data centers and offices. The commitment will make Google "the world's largest corporate buyer of renewable power, with commitments reaching 2.6 gigawatts (2,600 megawatts) of wind and solar energy".^{[129] [130] [131]}

Further reading

• IEEE Micro. 23. 22–28. March–April 2002. L.A. Barroso . J. Dean . U. Hölzle . Web search for a planet: The Google cluster architecture. 10.1109/MM.2003.1196112. 2.
• Shankland, Stephen, CNET news "Google uncloaks once-secret server ." April 1, 2009.

External links

• Google Research Publications
• Web Search for a Planet: The Google Cluster Architecture (Luiz André Barroso, Jeffrey Dean, Urs Hölzle)

Notes and References

1. Web site: How Many Servers Does Google Have? . Data Center Knowledge . March 16, 2017 . September 20, 2018.
2. Web site: Google data centers, locations. July 21, 2014.
3. Web site: ISO/IEC 27001 - Compliance . 2023-07-11 . Google Cloud . en.
4. Web site: Report . Times-Mirror Staff . Google 'caps off' $600M investment in Loudoun County . February 12, 2021 . LoudounTimes.com . June 19, 2019 . en.
5. Web site: Google Plans 2 Loudoun Data Centers . 12 April 2023 . LoudounNow.com . November 29, 2017 . en.
6. Web site: Arellano (CIPER) . Alberto . 2020-05-25 . Las zonas oscuras de la evaluación ambiental que autorizó "a ciegas" el megaproyecto de Google en Cerrillos . 2023-07-10 . CIPER Chile . es.
7. Web site: Google instalará un nuevo data center en Chile . dead . https://web.archive.org/web/20210922081126/http://www.diarioeldia.cl/tendencias/google-instalara-nuevo-data-center-en-chile . September 22, 2021 . September 28, 2019 . diarioeldia.cl . es.
8. Web site: 2021-12-01 . Una nueva Región de Google Cloud en Santiago, todo el potencial de la nube ahora más cerca . 2023-07-10 . Google.
9. Web site: A Google Cloud region now available in Columbus, Ohio . 2023-07-10 . Google Cloud Blog . en-US.
10. Web site: Namaste, India. Our new cloud region in Delhi NCR is now live. . cloudonair.withgoogle.com.
11. Web site: New Google Cloud region now open in Qatar . 2023-07-10 . Google Cloud Blog . en-US.
12. Web site: Dublin, Ireland – Data Centers – Google . April 2, 2019 . www.google.com.
13. Web site: PROJEKT IN HANAU: RECHENZENTRUM FÜR GOOGLE ENTWICKELT . NDC-GARBE. September 2, 2021 .
14. Web site: Google Cloud investing in Germany with new infrastructure and sustainable energy . 2023-07-10 . Google Cloud Blog . en-US.
15. Web site: 2018-11-20 . Breaking ground for Google's first data center in Denmark . 2023-07-10 . Google . en-us.
16. Web site: September 11, 2020 . Google gets green light for its sixth data center in Hamina, Finland .
17. Web site: Investing in Google infrastructure, investing in Nevada. . 2023-07-10 . Google Cloud Blog . en-US.
18. Web site: Torres-Cortez . Ricardo . September 16, 2020 . Google to invest additional $600M at Henderson data center – Las Vegas Sun Newspaper . February 12, 2021 . lasvegassun.com . en . "With this latest announcement, Google will bring their total investment in the city of Henderson to $1.2 billion," said Mayor Debra March in the release.
19. Web site: Baxtel . Google Henderson NV Data Center . February 12, 2021 . baxtel.com . en.
20. Web site: Coming soon: GCP's Hong Kong region . 2023-07-10 . Google Cloud Blog . en-US.
21. Web site: Growing our presence in Asia Pacific: New GCP regions in Hong Kong and Jakarta . 2023-07-10 . Google Cloud Blog . en-US.
22. Web site: The new Google Cloud region in Jakarta is now open . 2021-09-15 . Google Cloud Blog . en.
23. Web site: Google Cloud Platform Tokyo region now open for business . 2023-07-10 . Google Cloud Blog . en-US.
24. Web site: Google - Singapore . 2023-07-11 . www.streetdirectory.com.
25. Web site: Google Cloud Platform now open in London . 2023-07-10 . Google Cloud Blog . en-US.
26. Web site: New Google Cloud region in Madrid, Spain now open . 2023-07-10 . Google Cloud Blog . en-US.
27. Web site: The Google Cloud region in Melbourne is now open . 2023-07-10 . Google Cloud Blog . en-US.
28. Web site: Welcome . 2023-07-10 . Agriport A7 . nl-NL.
29. Web site: June 24, 2019 . Google to Spend $1.1 Billion on New Data Centers in Netherlands . Data Center Knowledge.
30. Web site: A Google Cloud region now available in Dallas, Texas . 2023-07-10 . Google Cloud Blog . en-US.
31. Web site: June 14, 2019 . Google's massive $600M data center takes shape in Ellis County as tech giant ups Texas presence . February 12, 2021 . Dallas News . en.
32. Web site: New Google Cloud region in Milan, Italy now open . 2023-07-10 . Google Cloud Blog . en-US.
33. Web site: Project to expand Google's activities in Quebec - Future computer data center in Beauharnois . May 10, 2021 . www.quebec.ca . fr.
34. Web site: GCP arrives in Canada with launch of Montréal region . 2023-07-10 . Google Cloud Blog . en-US.
35. Web site: Google's ongoing commitment in Quebec . February 12, 2021 . Google Blog - Company News . en.
36. Web site: Stiver . Dave . November 1, 2017 . GCP arrives in India with launch of Mumbai region . July 30, 2019 . Google Cloud Blog . en.
37. Web site: Williams . Mark . Google joins New Albany high-tech crowd with $600 million data center . February 12, 2021 . The Columbus Dispatch . en . December 30, 2020 . https://web.archive.org/web/20201230214959/https://www.dispatch.com/business/20191103/google-joins-new-albany-high-tech-crowd-with-600-million-data-center . dead .
38. Web site: New Albany, Ohio – Data Centers – Google . February 12, 2021 . Google Data Centers . en.
39. Web site: Google Cloud launches new Osaka region to support growing customer base in Japan . 2023-07-10 . Google Cloud Blog . en-US.
40. Web site: GCP arrives in South America with launch of São Paulo region! . 2023-07-10 . Google Cloud Blog . en-US.
41. Web site: Google confirms it is behind $600m Papillion data center project . February 12, 2021 . www.datacenterdynamics.com . en.
42. Web site: Papillion, Nebraska – Data Centers – Google . February 12, 2021 . Google Data Centers . en.
43. Web site: Google Cloud region in Paris France now open . 2023-07-10 . Google Cloud Blog . en-US.
44. Web site: Dawn-Hiscox . Tanwen . February 20, 2018 . Google to spend m on Pryor data center expansion . live . https://web.archive.org/web/20190423135640/https://www.datacenterdynamics.com/news/google-to-spend-600m-on-pryor-data-center-expansion/ . April 23, 2019 . April 23, 2019 . Data Centre Dynamics.
45. Web site: September 28, 2018 . Google ha decido de invertir millones de dólares en su centro de datos en Chile . dead . https://web.archive.org/web/20190330032359/http://newtechmag.net/2018/09/28/google-ha-decidido-invertir-140-millones-de-dolares-en-su-centro-de-datos-de-chile/ . March 30, 2019 . December 8, 2018 . Newtechmag.net . es.
46. Web site: Tanwen Dawn-Hiscox . April 18, 2017 . Google is planning a massive data center in Nevada . December 8, 2018 . Datacenterdynamics.com.
47. Web site: Jason Hidalgo . November 16, 2018 . Nevada approves Google's M data center near Las Vegas, M in tax incentives . December 8, 2018 . Rgj.com.
48. Web site: Jason Hidalgo . September 16, 2020 . Google to invest $600 million in data center near Reno, gets tax break . October 26, 2020 . Reno Gazette Journal . "With our new data center in Storey County and our expanded investment in our Henderson site, Google will have two facilities in Nevada, bringing our total investment to over $1.88 billion.".
49. Web site: Google Cloud region in Salt Lake City now open . 2023-07-10 . Google Cloud Blog . en-US.
50. Web site: New GCP Region in Seoul . 2023-07-10 . Google Cloud Blog . en-US.
51. Web site: Google Cloud expands to Australia with new Sydney region . 2023-07-10 . Google Cloud Blog . en-US.
52. Web site: Google launches GCP region in Tel Aviv, Israel . datacenterdynamics.com.
53. Web site: Google Cloud region in Tel Aviv Israel now open . 2023-07-10 . Google Cloud Blog . en-US.
54. Web site: Google Cloud Toronto region now open . 2023-07-10 . Google Cloud Blog . en-US.
55. Web site: New Google Cloud region in Turin, Italy now open . 2023-07-10 . Google Cloud Blog . en-US.
56. Web site: September 27, 2019 . Google to Build Cloud Data Centers in Poland . Data Center Knowledge.
57. Web site: Google kicks off construction on M Alabama data center . August 19, 2019 . Made in Alabama . April 9, 2018 . en.
58. Web site: Die Schweizer Google Cloud Platform zieht zu Green in den Aargau . 2023-07-10 . Swiss IT Magazine . de.
59. <
60. Web site: Introducing 5 new Google Cloud regions . 2023-07-10 . Google Cloud Blog . en-US.
61. News: 2022-09-30 . Google plant ein großes Rechenzentrum südlich des Flughafens BER . de . Der Spiegel . 2023-07-10 . 2195-1349.
62. Web site: Google Cloud investing in Germany with new infrastructure and sustainable energy . 2023-07-10 . Google Cloud Blog . en-US.
63. Web site: Berlin-Brandenburg Google Cloud region is now open . 2024-02-03 . Google Cloud Blog . en-US.
64. Web site: Google Cloud Platform region updates . 2023-07-10 . Google Cloud Blog . en-US.
65. Web site: Google affiliate's latest move signals selection of KC for $600M data center . February 12, 2021 . www.bizjournals.com.
66. Web site: Bringing a new Google Cloud region to Kuwait . 2023-07-10 . Google Cloud Blog . en-US.
67. Web site: Announcing new Google Cloud regions in Asia Pacific . 2023-07-10 . Google Cloud Blog . en-US.
68. Web site: Announcing a new Google Cloud region in Mexico . 2023-07-10 . Google Cloud Blog . en-US.
69. Web site: September 12, 2019 . Google purchases land for new data center in Tainan . December 20, 2019 . Taipei Times.
70. Web site: September 11, 2019 . Google to set up data center in Tainan . December 20, 2019 . Focus Taiwan.
71. Web site: September 11, 2019 . Google to set up second data center in Taiwan . December 20, 2019 . Taiwan News.
72. Web site: September 3, 2020 . Google confirms plans to build 3rd data center in Taiwan . September 3, 2020 . Taiwan News.
73. Web site: 2024-01-18 . Our $1 billion investment in a new UK data centre . 2024-01-19 . Google . en-us.
74. Web site: "Google Stanford Hardware" . March 23, 2017 . bot: unknown . https://web.archive.org/web/19990209043945/http://google.stanford.edu/googlehardware.html . February 9, 1999 . . Stanford University (provided by Internet Archive). Retrieved on July 10, 2006.
75. Web site: Case Study: Live upgrading many thousand of servers from an ancient Red Hat distribution to a 10 year newer Debian based one. Merlin. Marc. 2013. Linux Foundation. June 9, 2017.
76. Book: Strategies for E-business. Tawfik Jelassi . Albrecht Enders . Case study 16 — Google. Pearson Education. 2004. 0-273-68840-5-->,978-0-273-68840-2. 424.
77. Computer Architecture, Fifth Edition: A Quantitative Approach, ; Chapter Six; 6.7 "A Google Warehouse-Scale Computer" page 471 "Designing motherboards that only need a single 12-volt supply so that the UPS function could be supplied by standard batteries associated with each server"
78. https://www.cnet.com/news/google-uncloaks-once-secret-server-10209580/ Google uncloaks once-secret server
79. Web site: Google Sustainability. Google Sustainability.
80. Web site: Analytics Press Growth in data center electricity use 2005 to 2010 . May 22, 2012 . https://web.archive.org/web/20120111082558/http://www.analyticspress.com/datacenters.html . January 11, 2012 . dead .
81. http://blogs.nmscommunications.com/communications/2008/05/google-surpasses-supercomputer-community-unnoticed.html Google Surpasses Supercomputer Community, Unnoticed?
82. http://static.googleusercontent.com/external_content/untrusted_dlcp/research.google.com/en/us/archive/googlecluster-ieee.pdf Web Search for a Planet: The Google Cluster Architecture
83. Web site: Warehouse size computers . May 22, 2012 . June 19, 2012 . https://web.archive.org/web/20120619202125/http://bnrg.eecs.berkeley.edu/~randy/Courses/CS294.F09/wharehousesizedcomputers.pdf . dead .
84. http://research.google.com/pubs/archive/37069.pdf Denis Abt High Performance Datacenter Networks: Architectures, Algorithms, and Opportunities
85. Book: Network Programming in .NET. Fiach Reid. 251–253. Case Study: The Google search engine. Digital Press. 2004. 978-1-55558-315-6. https://archive.org/details/networkprogrammi00fiac/page/251.
86. Web site: Google Data Center FAQ . Rich Miller . Data Center Knowledge . March 27, 2008 . March 15, 2009 . https://web.archive.org/web/20090313082216/http://www.datacenterknowledge.com/archives/2008/03/27/google-data-center-faq/ . March 13, 2009 . dead . mdy-all .
87. Web site: Found: Google Australia's secret data network . Brett Winterford. ITNews . March 5, 2010 . March 20, 2010.
88. Book: Singh. Arjun. Ong. Joon. Agarwal. Amit. Anderson. Glen. Armistead. Ashby. Bannon. Roy. Boving. Seb. Desai. Gaurav. Felderman. Bob. Germano. Paulie. Kanagala. Anand. Proceedings of the 2015 ACM Conference on Special Interest Group on Data Communication . Jupiter Rising: A Decade of Clos Topologies and Centralized Control in Google's Datacenter Network . 2015 . 183–197 . https://research.google/pubs/pub43837/ . 10.1145/2785956.2787508. 2817692. free. 978-1-4503-3542-3 .
89. Markoff, John; Hansell, Saul. "Hiding in Plain Sight, Google Seeks More Power." New York Times. June 14, 2006. Retrieved on October 15, 2008.
90. Google "The Dalles, Oregon Data Center" Retrieved on January 3, 2011.
91. Strand, Ginger."Google Data Center" Harper's Magazine. March 2008. Retrieved on October 15, 2008.
92. Web site: Stora Enso divests Summa Mill premises in Finland for million . . February 12, 2009 . December 2, 2009 . https://web.archive.org/web/20090413023322/http://www.storaenso.com/media-centre/press-releases/2009/02/Pages/stora-enso-divests-summa-mill.aspx . April 13, 2009 . dead .
93. Stooora yllätys: Google ostaa Summan tehtaan . . February 12, 2009 . Helsinki . February 12, 2009 . fi . https://web.archive.org/web/20090214232026/http://www.kauppalehti.fi/5/i/talous/uutiset/etusivu/uutinen.jsp?oid=2009%2F02%2F18987 . February 14, 2009 . dead .
94. Google investoi 200 miljoonaa euroa Haminaan . . February 4, 2009 . Helsinki . March 15, 2009 . fi.
95. Web site: Hamina, Finland. April 23, 2018.
96. http://www.fincloud.freehostingcloud.com/ Finland – First Choice for Siting Your Cloud Computing Data Center.
97. Web site: Google's worst-kept secret: floating data centers off US coasts. Theguardian.com. October 30, 2013. Rory Carroll. December 8, 2018.
98. Web site: Google Gets Patent for Data Center Barges. Datacenterknowledge.com. April 29, 2009. Rich Miller. December 8, 2018.
99. Web site: Google files patent for wave-powered floating data center. Cnet.com. September 8, 2008. Martin Lamonica. December 8, 2018.
100. Web site: Google's ship based datacenter patent application surfaces. Datacenterdynamics.com. September 7, 2008. December 8, 2018.
101. Web site: Google barge mystery solved: they're for 'interactive learning centers'. Theguardian.com. November 6, 2013. December 8, 2018.
102. News: Google confirms selling a mystery barge. August 1, 2014. San Jose Mercury News. Brandon Bailey. April 7, 2015.
103. News: What Happened To Those Google Barges?. November 7, 2014. Consumerist. Chris Morran. January 15, 2017.
104. Book: An Introduction to Search Engines and Web Navigation. Mark Levene. Pearson Education . 2005. 0-321-30677-5-->,978-0-321-30677-7. 73.
105. Web site: Python Status Update . Artima . January 10, 2006 . February 17, 2012.
106. Web site: Warning . Panela . Blog-city . February 17, 2012 . dead . https://web.archive.org/web/20111228034932/http://panela.blog-city.com/python_at_google_greg_stein__sdforum.htm . December 28, 2011 .
107. Web site: Quotes about Python . Python . February 17, 2012.
108. Web site: Google Architecture . High Scalability . November 22, 2008 . February 17, 2012.
109. Web site: Colossus: Successor to the Google File System (GFS) . SysTutorials . November 29, 2012 . May 10, 2016.
110. Web site: 2008-07-28. Google alums rev up a new search engine. 2021-09-15. Los Angeles Times. en-US.
111. Web site: Google official release note . September 28, 2013.
112. Web site: Google Developing Caffeine Storage System | TechWeekEurope UK . Eweekeurope.co.uk . August 18, 2009 . February 17, 2012 . https://web.archive.org/web/20111115162704/http://www.eweekeurope.co.uk/news/news-it-infrastructure/google-developing-caffeine-storage-system-1620 . November 15, 2011 . dead .
113. Web site: Developer Guide – Protocol Buffers – Google Code . February 17, 2012.
114. http://static.googleusercontent.com/media/research.google.com/en/us/archive/bigtable-osdi06.pdf
115. Web site: windley on . Phil Windley's Technometria | Velocity 08: Storage at Scale . Windley.com . June 24, 2008 . February 17, 2012.
116. Web site: Message limit – Protocol Buffers | Google Groups . February 17, 2012.
117. Web site: Jeff Dean's keynote at WSDM 2009 . February 17, 2012.
118. Daniel Peng, Frank Dabek. (2010). Large-scale Incremental Processing Using Distributed Transactions and Notifications. Proceedings of the 9th USENIX Symposium on Operating Systems Design and Implementation.
119. The Register. Google Caffeine jolts worldwide search machine
120. The Register. Google Percolator – global search jolt sans MapReduce comedown
121. Book: Future of Google Earth. Chandler Evans. Google Platform. Madison Publishing Company. 2008. 1-4196-8903-7-->,978-1-4196-8903-1. 299.
122. Book: Google Power. Chris Sherman. 10–11. McGraw-Hill Professional. 2005. 0-07-225787-3-->,978-0-07-225787-8. How Google Works.
123. Book: How Google Works. 17–18. Googlepedia. Michael Miller. Pearson Technology Group. 2007. 0-7897-3639-X-->,978-0-7897-3639-0.
124. News: NSA infiltrates links to Yahoo, Google data centers worldwide, Snowden documents say. Gellman. Barton. October 30, 2013. The Washington Post. November 1, 2013. Soltani. Ashkan.
125. Web site: N.S.A. Said to Tap Google and Yahoo Abroad. Savage. Charlie. Miller. Claire Cain. October 30, 2013. The New York Times. March 9, 2017. Perlroth. Nicole.
126. Web site: How the NSA's MUSCULAR tapped Google's and Yahoo's private networks. Gallagher. Sean. October 31, 2013. Ars Technica. Condé Nast. March 9, 2017.
127. Web site: Angry Over U.S. Surveillance, Tech Giants Bolster Defenses. Miller. Claire Cain. October 31, 2013. The New York Times. March 9, 2017.
128. Web site: Google's most efficient data center runs at 95 degrees. Humphries. Matthew. March 27, 2012. geek.com. https://web.archive.org/web/20160613112324/http://www.geek.com/chips/googles-most-efficient-data-center-runs-at-95-degrees-1478473/. June 13, 2016. June 13, 2016.
129. Web site: We're set to reach 100% renewable energy — and it's just the beginning. Hölzle. Urs. December 6, 2016. The Keyword Google Blog. December 8, 2016.
130. Web site: Google just notched a big victory in the fight against climate change. Statt. Nick. December 6, 2016. The Verge. Vox Media. December 8, 2016.
131. Web site: Google says it will hit 100% renewable energy by 2017. Etherington. Darrell. December 7, 2016. TechCrunch. AOL. December 8, 2016.