AMD 64 manual

P erf ormance Guidelines f or AMD Athlon™ 64 and AMD Opter on™ ccNUMA Multipr ocessor Systems Application Note 40555 Publication # Revision: 3.00 June 2006 Issue Date:[...]

© 2006 Advanced Micr o Devices, Inc. All rights reserv ed. The contents of this document are provided in connection with Advanced Micro De vices, Inc. (“ AMD”) products. AMD mak es no represen tations or w arranties with respect to the accuracy or completeness of the contents of this publication and reserv es the right to make changes to speci[...]

Contents 3 Performance Guidelines for AMD Athlon™ 64 and AMD Opter on™ ccNUMA Multipr o cessor Systems 40555 Rev . 3.00 June 2006 Contents Revision History . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7 Chapter 1 Introduction . . . . . . . . . . . . . . . . . . [...]

4 Contents 40555 Rev . 3.00 June 2006 Performance Guidelines for AMD Athlon™ 64 an d AMD Opter on™ ccNUMA Multipr ocessor Systems A.2.1 What Resources Are Used When a Single Read-Only or Write-Only Thread Accesses Remote Data? . . . . . . . . . . . . . . . . . . . . . . . . . .40 A.2.2 What Resources Are Used Wh en Two Write-only Threads Fire a[...]

List of Figur es 5 Performance Guidelines for AMD Athlon™ 64 and AMD Opter on™ ccNUMA Multipr o cessor Systems 40555 Rev . 3.00 June 2006 List of Figures Figure 1. Quartet Topology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .14 Figure 2. Internal Resources Associated with a Quartet [...]

6 List of Figur es 40555 Rev . 3.00 June 2006 Performance Guidelines for AMD Athlon™ 64 an d AMD Opter on™ ccNUMA Multipr ocessor Systems[...]

Revision History 7 Performance Guidelines for AMD Athlon™ 64 and AMD Opter on™ ccNUMA Multipr o cessor Systems 40555 Rev . 3.00 June 2006 Re vision Histor y Date Revision Descript ion June 20 06 3.00 Initial release .[...]

8 Revision History 40555 Rev . 3.00 June 2006 Performance Guidelines for AMD Athlon™ 64 an d AMD Opter on™ ccNUMA Multipr ocessor Systems[...]

Chapter 1 Intr oduction 9 Performance Guidelines for AMD Athlon™ 64 and AMD Opter on™ ccNUMA Multipr o cessor Systems 40555 Rev . 3.00 June 2006 Chapter 1 Intr oduction The AMD Athlon™ 64 and AMD Opteron™ family of single-core and dual-core multiprocessor systems are based on the cache coherent Non-Un iform Memory Access (ccNUM A) architect[...]

10 Intr oduction Chapter 1 40555 Rev . 3.00 June 2006 Performance Guidelines for AMD Athlon™ 64 an d AMD Opter on™ ccNUMA Multipr ocessor Systems bandwidth test, it ex ercises both of these modes of operation. The test serv es as a latency sensiti ve test case when the test threads perfor m read-only operations and as a bandw idth sensiti ve te[...]

Chapter 1 Intr oduction 11 Performance Guidelines for AMD Athlon™ 64 and AMD Opter on™ ccNUMA Multipr o cessor Systems 40555 Rev . 3.00 June 2006 [12] http://msdn.microsoft.com/library/default.asp?url=/library/ en-us/dngenlib/html/ msdn_heapmm.asp [13] http://msdn.microsoft.com/library/default.asp?url=/library/en-us/memory/base/ lo w_fragmentat[...]

12 Intr oduction Chapter 1 40555 Rev . 3.00 June 2006 Performance Guidelines for AMD Athlon™ 64 an d AMD Opter on™ ccNUMA Multipr ocessor Systems[...]

Chapter 2 Experimental Setup 13 Performance Guidelines for AMD Athlon™ 64 and AMD Opter on™ ccNUMA Multipr o cessor Systems 40555 Rev . 3.00 June 2006 Chapter 2 Experimental Setup This chapter presents a description of the experi mental en vironment within which the following performance study was carried out. This section describes the ha rdw [...]

14 Experimental Setup Chapter 2 40555 Rev . 3.00 June 2006 Performance Guidelines for AMD Athlon™ 64 an d AMD Opter on™ ccNUMA Multipr ocessor Systems Figure 1. Quartet T opolog y The term hop is commonly used to describe access dist ances on NUMA systems. When a thread accesses memory on the same node as that on which it is r unning, it is a 0[...]

Chapter 2 Experimental Setup 15 Performance Guidelines for AMD Athlon™ 64 and AMD Opter on™ ccNUMA Multipr o cessor Systems 40555 Rev . 3.00 June 2006 Figure 2. Internal R esources Associated wi th a Quartet Node From the perspecti ve of the MCT , a memory request may come from either the local core or from another core ov er a coherent HyperT [...]

16 Experimental Setup Chapter 2 40555 Rev . 3.00 June 2006 Performance Guidelines for AMD Athlon™ 64 an d AMD Opter on™ ccNUMA Multipr ocessor Systems resources approach saturation. The test has two modes: r ead-only and write-only . When the test threads are read-only , the throughput does not stress the capacity of the system resources and, t[...]

Chapter 2 Experimental Setup 17 Performance Guidelines for AMD Athlon™ 64 and AMD Opter on™ ccNUMA Multipr o cessor Systems 40555 Rev . 3.00 June 2006 characterization of the resource beha vior in the system. These recommendati ons, coupled with these interesting cases, provide an understandin g of the lo w-le vel beha vior of the system, which[...]

18 Experimental Setup Chapter 2 40555 Rev . 3.00 June 2006 Performance Guidelines for AMD Athlon™ 64 an d AMD Opter on™ ccNUMA Multipr ocessor Systems 2.3.2 Labels Used Each of the bars on the graph is labele d with the hop information for the thread. 2.3.3 Y-Axis Displa y For the one-thread test cases on the id le system, the graphs sho w th e[...]

Chapter 3 Analysis and Recommendations 19 Performance Guidelines for AMD Athlon™ 64 and AMD Opter on™ ccNUMA Multipr o cessor Systems 40555 Rev . 3.00 June 2006 Chapter 3 Anal ysis and Recommendations This section lays out recommendations to de ve lopers. Se veral of these recommendations are accompanied b y empirical results collected fr om te[...]

20 Analysis and Recommendations Chapter 3 40555 Rev . 3.00 June 2006 Performance Guidelines for AMD Athlon™ 64 an d AMD Opter on™ ccNUMA Multipr ocessor Systems 3.1.2 Multiple Threads-Shared Data When scheduling multiple threads that share data on an idle system, it is preferable to schedule the threads on both cores of an idle node f irst, the[...]

Chapter 3 Analysis and Recommendations 21 Performance Guidelines for AMD Athlon™ 64 and AMD Opter on™ ccNUMA Multipr o cessor Systems 40555 Rev . 3.00 June 2006 distance. If they are indirectly connected to each ot her in a 4P conf iguration, it is considered as a 2 hop access distance. The follo wing example—extracted from mining the re sult[...]

22 Analysis and Recommendations Chapter 3 40555 Rev . 3.00 June 2006 Performance Guidelines for AMD Athlon™ 64 an d AMD Opter on™ ccNUMA Multipr ocessor Systems Figure 5. Write-Only Thread Running on No de 0, Accessing Data from 0, 1 and 2 Hops A wa y on an Idle System In this test case, a write access is similar to a read acc ess as far as the[...]

Chapter 3 Analysis and Recommendations 23 Performance Guidelines for AMD Athlon™ 64 and AMD Opter on™ ccNUMA Multipr o cessor Systems 40555 Rev . 3.00 June 2006 A ccNUMA-aw are OS keeps data local on the node wh ere f irst-touch occurs as long as there is enough physical memory a vailable on that node. If enough physical memory is not a vailabl[...]

24 Analysis and Recommendations Chapter 3 40555 Rev . 3.00 June 2006 Performance Guidelines for AMD Athlon™ 64 an d AMD Opter on™ ccNUMA Multipr ocessor Systems afterwords no longer needs the data structure and if only one of the work er threads needs the data structure. In other words, the data structure is not truly shared between the work er[...]

Chapter 3 Analysis and Recommendations 25 Performance Guidelines for AMD Athlon™ 64 and AMD Opter on™ ccNUMA Multipr o cessor Systems 40555 Rev . 3.00 June 2006 Spec JBB 2005 was run using the NUMA tools pro vided by Linux ® to measu re the performance improv ement with node interlea vi ng. The results were obtained on the same internal 4P Qua[...]

26 Analysis and Recommendations Chapter 3 40555 Rev . 3.00 June 2006 Performance Guidelines for AMD Athlon™ 64 an d AMD Opter on™ ccNUMA Multipr ocessor Systems • Threads firing at each other ( cr ossfire ) The f irst thread runs on node 0 and writes to me mory on node 1 (1 hop). The second thread runs on node 1 and writes to memory on node 0[...]

Chapter 3 Analysis and Recommendations 27 Performance Guidelines for AMD Athlon™ 64 and AMD Opter on™ ccNUMA Multipr o cessor Systems 40555 Rev . 3.00 June 2006 Here the same two fore ground threads as before were run t hough the cases as before— local , cr ossfire and no cr ossfir e . In addition, four background threads are left running on:[...]

28 Analysis and Recommendations Chapter 3 40555 Rev . 3.00 June 2006 Performance Guidelines for AMD Athlon™ 64 an d AMD Opter on™ ccNUMA Multipr ocessor Systems Figure 8. Crossfire 1 Hop-1 Hop Case vs No Cr ossfire 1 Hop-1 Hop Case under a V ery High Backgr ound Lo ad (High Subscription) Next, we increase the number of ba ckground threads to si[...]

Chapter 3 Analysis and Recommendations 29 Performance Guidelines for AMD Athlon™ 64 and AMD Opter on™ ccNUMA Multipr o cessor Systems 40555 Rev . 3.00 June 2006 Figure 9. Crossfire 1 Hop-1 Hop Case vs No Cr ossfire 1 Hop-1 Hop Case under a V ery High Backgr ound Lo ad (Full Subscription) In the no crossf ire case, the tota l memory bandwidth ob[...]

30 Analysis and Recommendations Chapter 3 40555 Rev . 3.00 June 2006 Performance Guidelines for AMD Athlon™ 64 an d AMD Opter on™ ccNUMA Multipr ocessor Systems This analogy clearly communicates the performance ef fects of queuing time versus latency . In a computer serv er , with ma ny concurrent outstanding memory reque sts, we would gladly i[...]

Chapter 3 Analysis and Recommendations 31 Performance Guidelines for AMD Athlon™ 64 and AMD Opter on™ ccNUMA Multipr o cessor Systems 40555 Rev . 3.00 June 2006 Ho wev e r , as shown in Figure 11 on page 31, when both threads are write-only , the 0 hop-1 hop and 0 hop-2 hop cases are faster than the 0 hop-0 hop case. Figure 11. Both Write-Onl y[...]

32 Analysis and Recommendations Chapter 3 40555 Rev . 3.00 June 2006 Performance Guidelines for AMD Athlon™ 64 an d AMD Opter on™ ccNUMA Multipr ocessor Systems In addition, three background thre ads are running on nodes 1, 2 and 3. Each of these background threads access data loca lly . The rate of memo ry demand b y each these threads is v ar[...]

Chapter 3 Analysis and Recommendations 33 Performance Guidelines for AMD Athlon™ 64 and AMD Opter on™ ccNUMA Multipr o cessor Systems 40555 Rev . 3.00 June 2006 Figure 13. Both Write-Only Threads Running on Node 0 (Diff erent Cores) under Medium Bac kground Load (High Subscription) Figure 14. Both Write-Only Threads Running on Node 0 (Dif feren[...]

34 Analysis and Recommendations Chapter 3 40555 Rev . 3.00 June 2006 Performance Guidelines for AMD Athlon™ 64 an d AMD Opter on™ ccNUMA Multipr ocessor Systems Figure 15. Both Write-Only Threads Running on Node 0 (Different Cores) under V ery High Backgr ound Load (H igh Subscription) Under a very high background load, for the 0 hop-1 hop case[...]

Chapter 3 Analysis and Recommendations 35 Performance Guidelines for AMD Athlon™ 64 and AMD Opter on™ ccNUMA Multipr o cessor Systems 40555 Rev . 3.00 June 2006 3.6 P arallelism Exposed by Compiler s on AMD ccNUMA Multipr ocessor Systems Se veral compilers for AMD multiprocessor system s pro vide additional hooks to allow automatic parallelizat[...]

36 Analysis and Recommendations Chapter 3 40555 Rev . 3.00 June 2006 Performance Guidelines for AMD Athlon™ 64 an d AMD Opter on™ ccNUMA Multipr ocessor Systems[...]

Chapter 4 Conclusions 37 Performance Guidelines for AMD Athlon™ 64 and AMD Opter on™ ccNUMA Multipr o cessor Systems 40555 Rev . 3.00 June 2006 Chapter 4 Conc lusions The single most important recommendation for most applications is to keep data local on node where it is being accessed. As long as a thread initializes the data it need s, in oth[...]

38 Conclusions Chapter 4 40555 Rev . 3.00 June 2006 Performance Guidelines for AMD Athlon™ 64 an d AMD Opter on™ ccNUMA Multipr ocessor Systems Data placement tools can also come in handy when a thread needs more data than the amount of physical memory a vailable o n a node . Certain OSs also allo w data migration wi th these tools or API. Usin[...]

Appendix A 39 Performance Guidelines for AMD Athlon™ 64 and AMD Opter on™ ccNUMA Multipr o cessor Systems 40555 Rev . 3.00 June 2006 Appendix A The follo wing sections provide additional e xplanator y information on topics disc ussed in the pre vious sections of this document. A.1 Description of the Buffer Queues Figure 16 sho ws the internal r[...]

40 Appendix A 40555 Rev . 3.00 June 2006 Performance Guidelines for AMD Athlon™ 64 an d AMD Opter on™ ccNUMA Multipr ocessor Systems Like wise packets to be transmitted from the MCT to the XB ar are queued in the “MCT -to-XBar” buf fers. The buf f ers in the SRI, XBar and MCT can be vie wed as staggered queues on the v arious units. A.2 Wh [...]

Appendix A 41 Performance Guidelines for AMD Athlon™ 64 and AMD Opter on™ ccNUMA Multipr o cessor Systems 40555 Rev . 3.00 June 2006 4.4 GB/s necessary . The two coherent HyperT ranspor t links are loaded at 3.5 GB/s each. Thus the utilization of each of the two co herent HyperT ransport links that connect node 0 and node 1 equals 87% (3.5÷4).[...]

42 Appendix A 40555 Rev . 3.00 June 2006 Performance Guidelines for AMD Athlon™ 64 an d AMD Opter on™ ccNUMA Multipr ocessor Systems A.3 Wh y Is the No Cr ossfire Case Slo wer Than the Cr ossfire Case on a System under a V er y High Bac kgr ound Load (Full Subscription)? When the threads are f iring at each other (crossfire) and all other free [...]

Appendix A 43 Performance Guidelines for AMD Athlon™ 64 and AMD Opter on™ ccNUMA Multipr o cessor Systems 40555 Rev . 3.00 June 2006 A.5 Wh y Is 0 Hop-1 Hop Case Slo wer Than 0 Hop-0 Hop Case on a System under High Bac kgr ound Load (High Subscription) for Write- Onl y Threads? When a 0 hop-0 hop scenario is subjected to a very high back ground[...]

44 Appendix A 40555 Rev . 3.00 June 2006 Performance Guidelines for AMD Athlon™ 64 an d AMD Opter on™ ccNUMA Multipr ocessor Systems A.7 T ools and APIs f or Thread/Pr ocess and Memory Placement (Affinity) f or AMD64 ccNUMA Multipr ocessor Systems This follo wing sections discuss tools and APIs av ailable for as signing thread/process and memor[...]

Appendix A 45 Performance Guidelines for AMD Athlon™ 64 and AMD Opter on™ ccNUMA Multipr o cessor Systems 40555 Rev . 3.00 June 2006 Controlling Memory Affinity Both numactl and libnuma library functions can be u sed to set memory af f inity[ 5 ]. Memory af finity set by tools lik e numactl applies to all the data accessed by th e entire progra[...]

46 Appendix A 40555 Rev . 3.00 June 2006 Performance Guidelines for AMD Athlon™ 64 an d AMD Opter on™ ccNUMA Multipr ocessor Systems The function to set memory af finity for a thread is V irtualAlloc( ) [ 9 ]. This function giv es the de veloper the choice to bind memory immediately on allocation or to defer binding until f irst touch. Although[...]

Appendix A 47 Performance Guidelines for AMD Athlon™ 64 and AMD Opter on™ ccNUMA Multipr o cessor Systems 40555 Rev . 3.00 June 2006 A.8.4 Node Interlea ving Configuration in the BIOS AMD Opteron™ and Athlon™ 64 ccNUMA multiprocessor systems can be configured in the BIOS to interleav e all memory across all nodes on a pa ge basis (4KB for r[...]

48 Appendix A 40555 Rev . 3.00 June 2006 Performance Guidelines for AMD Athlon™ 64 an d AMD Opter on™ ccNUMA Multipr ocessor Systems[...]