| Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 1 | Dynamic DMA mapping | 
|  | 2 | =================== | 
|  | 3 |  | 
|  | 4 | David S. Miller <davem@redhat.com> | 
|  | 5 | Richard Henderson <rth@cygnus.com> | 
|  | 6 | Jakub Jelinek <jakub@redhat.com> | 
|  | 7 |  | 
|  | 8 | This document describes the DMA mapping system in terms of the pci_ | 
|  | 9 | API.  For a similar API that works for generic devices, see | 
|  | 10 | DMA-API.txt. | 
|  | 11 |  | 
|  | 12 | Most of the 64bit platforms have special hardware that translates bus | 
|  | 13 | addresses (DMA addresses) into physical addresses.  This is similar to | 
|  | 14 | how page tables and/or a TLB translates virtual addresses to physical | 
|  | 15 | addresses on a CPU.  This is needed so that e.g. PCI devices can | 
|  | 16 | access with a Single Address Cycle (32bit DMA address) any page in the | 
|  | 17 | 64bit physical address space.  Previously in Linux those 64bit | 
|  | 18 | platforms had to set artificial limits on the maximum RAM size in the | 
|  | 19 | system, so that the virt_to_bus() static scheme works (the DMA address | 
|  | 20 | translation tables were simply filled on bootup to map each bus | 
|  | 21 | address to the physical page __pa(bus_to_virt())). | 
|  | 22 |  | 
|  | 23 | So that Linux can use the dynamic DMA mapping, it needs some help from the | 
|  | 24 | drivers, namely it has to take into account that DMA addresses should be | 
|  | 25 | mapped only for the time they are actually used and unmapped after the DMA | 
|  | 26 | transfer. | 
|  | 27 |  | 
|  | 28 | The following API will work of course even on platforms where no such | 
|  | 29 | hardware exists, see e.g. include/asm-i386/pci.h for how it is implemented on | 
|  | 30 | top of the virt_to_bus interface. | 
|  | 31 |  | 
|  | 32 | First of all, you should make sure | 
|  | 33 |  | 
|  | 34 | #include <linux/pci.h> | 
|  | 35 |  | 
|  | 36 | is in your driver. This file will obtain for you the definition of the | 
|  | 37 | dma_addr_t (which can hold any valid DMA address for the platform) | 
|  | 38 | type which should be used everywhere you hold a DMA (bus) address | 
|  | 39 | returned from the DMA mapping functions. | 
|  | 40 |  | 
|  | 41 | What memory is DMA'able? | 
|  | 42 |  | 
|  | 43 | The first piece of information you must know is what kernel memory can | 
|  | 44 | be used with the DMA mapping facilities.  There has been an unwritten | 
|  | 45 | set of rules regarding this, and this text is an attempt to finally | 
|  | 46 | write them down. | 
|  | 47 |  | 
|  | 48 | If you acquired your memory via the page allocator | 
|  | 49 | (i.e. __get_free_page*()) or the generic memory allocators | 
|  | 50 | (i.e. kmalloc() or kmem_cache_alloc()) then you may DMA to/from | 
|  | 51 | that memory using the addresses returned from those routines. | 
|  | 52 |  | 
|  | 53 | This means specifically that you may _not_ use the memory/addresses | 
|  | 54 | returned from vmalloc() for DMA.  It is possible to DMA to the | 
|  | 55 | _underlying_ memory mapped into a vmalloc() area, but this requires | 
|  | 56 | walking page tables to get the physical addresses, and then | 
|  | 57 | translating each of those pages back to a kernel address using | 
|  | 58 | something like __va().  [ EDIT: Update this when we integrate | 
|  | 59 | Gerd Knorr's generic code which does this. ] | 
|  | 60 |  | 
| David Brownell | 21440d3 | 2006-04-01 10:21:52 -0800 | [diff] [blame] | 61 | This rule also means that you may use neither kernel image addresses | 
|  | 62 | (items in data/text/bss segments), nor module image addresses, nor | 
|  | 63 | stack addresses for DMA.  These could all be mapped somewhere entirely | 
|  | 64 | different than the rest of physical memory.  Even if those classes of | 
|  | 65 | memory could physically work with DMA, you'd need to ensure the I/O | 
|  | 66 | buffers were cacheline-aligned.  Without that, you'd see cacheline | 
|  | 67 | sharing problems (data corruption) on CPUs with DMA-incoherent caches. | 
|  | 68 | (The CPU could write to one word, DMA would write to a different one | 
|  | 69 | in the same cache line, and one of them could be overwritten.) | 
| Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 70 |  | 
|  | 71 | Also, this means that you cannot take the return of a kmap() | 
|  | 72 | call and DMA to/from that.  This is similar to vmalloc(). | 
|  | 73 |  | 
|  | 74 | What about block I/O and networking buffers?  The block I/O and | 
|  | 75 | networking subsystems make sure that the buffers they use are valid | 
|  | 76 | for you to DMA from/to. | 
|  | 77 |  | 
|  | 78 | DMA addressing limitations | 
|  | 79 |  | 
|  | 80 | Does your device have any DMA addressing limitations?  For example, is | 
|  | 81 | your device only capable of driving the low order 24-bits of address | 
|  | 82 | on the PCI bus for SAC DMA transfers?  If so, you need to inform the | 
|  | 83 | PCI layer of this fact. | 
|  | 84 |  | 
|  | 85 | By default, the kernel assumes that your device can address the full | 
|  | 86 | 32-bits in a SAC cycle.  For a 64-bit DAC capable device, this needs | 
|  | 87 | to be increased.  And for a device with limitations, as discussed in | 
|  | 88 | the previous paragraph, it needs to be decreased. | 
|  | 89 |  | 
|  | 90 | pci_alloc_consistent() by default will return 32-bit DMA addresses. | 
|  | 91 | PCI-X specification requires PCI-X devices to support 64-bit | 
|  | 92 | addressing (DAC) for all transactions. And at least one platform (SGI | 
|  | 93 | SN2) requires 64-bit consistent allocations to operate correctly when | 
|  | 94 | the IO bus is in PCI-X mode. Therefore, like with pci_set_dma_mask(), | 
|  | 95 | it's good practice to call pci_set_consistent_dma_mask() to set the | 
|  | 96 | appropriate mask even if your device only supports 32-bit DMA | 
|  | 97 | (default) and especially if it's a PCI-X device. | 
|  | 98 |  | 
|  | 99 | For correct operation, you must interrogate the PCI layer in your | 
|  | 100 | device probe routine to see if the PCI controller on the machine can | 
|  | 101 | properly support the DMA addressing limitation your device has.  It is | 
|  | 102 | good style to do this even if your device holds the default setting, | 
|  | 103 | because this shows that you did think about these issues wrt. your | 
|  | 104 | device. | 
|  | 105 |  | 
|  | 106 | The query is performed via a call to pci_set_dma_mask(): | 
|  | 107 |  | 
|  | 108 | int pci_set_dma_mask(struct pci_dev *pdev, u64 device_mask); | 
|  | 109 |  | 
| Paolo Ornati | 670e9f3 | 2006-10-03 22:57:56 +0200 | [diff] [blame] | 110 | The query for consistent allocations is performed via a call to | 
| Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 111 | pci_set_consistent_dma_mask(): | 
|  | 112 |  | 
|  | 113 | int pci_set_consistent_dma_mask(struct pci_dev *pdev, u64 device_mask); | 
|  | 114 |  | 
|  | 115 | Here, pdev is a pointer to the PCI device struct of your device, and | 
|  | 116 | device_mask is a bit mask describing which bits of a PCI address your | 
|  | 117 | device supports.  It returns zero if your card can perform DMA | 
|  | 118 | properly on the machine given the address mask you provided. | 
|  | 119 |  | 
| Matt LaPlante | 84eb8d0 | 2006-10-03 22:53:09 +0200 | [diff] [blame] | 120 | If it returns non-zero, your device cannot perform DMA properly on | 
| Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 121 | this platform, and attempting to do so will result in undefined | 
|  | 122 | behavior.  You must either use a different mask, or not use DMA. | 
|  | 123 |  | 
|  | 124 | This means that in the failure case, you have three options: | 
|  | 125 |  | 
|  | 126 | 1) Use another DMA mask, if possible (see below). | 
|  | 127 | 2) Use some non-DMA mode for data transfer, if possible. | 
|  | 128 | 3) Ignore this device and do not initialize it. | 
|  | 129 |  | 
|  | 130 | It is recommended that your driver print a kernel KERN_WARNING message | 
|  | 131 | when you end up performing either #2 or #3.  In this manner, if a user | 
|  | 132 | of your driver reports that performance is bad or that the device is not | 
|  | 133 | even detected, you can ask them for the kernel messages to find out | 
|  | 134 | exactly why. | 
|  | 135 |  | 
|  | 136 | The standard 32-bit addressing PCI device would do something like | 
|  | 137 | this: | 
|  | 138 |  | 
|  | 139 | if (pci_set_dma_mask(pdev, DMA_32BIT_MASK)) { | 
|  | 140 | printk(KERN_WARNING | 
|  | 141 | "mydev: No suitable DMA available.\n"); | 
|  | 142 | goto ignore_this_device; | 
|  | 143 | } | 
|  | 144 |  | 
|  | 145 | Another common scenario is a 64-bit capable device.  The approach | 
|  | 146 | here is to try for 64-bit DAC addressing, but back down to a | 
|  | 147 | 32-bit mask should that fail.  The PCI platform code may fail the | 
|  | 148 | 64-bit mask not because the platform is not capable of 64-bit | 
|  | 149 | addressing.  Rather, it may fail in this case simply because | 
|  | 150 | 32-bit SAC addressing is done more efficiently than DAC addressing. | 
|  | 151 | Sparc64 is one platform which behaves in this way. | 
|  | 152 |  | 
|  | 153 | Here is how you would handle a 64-bit capable device which can drive | 
|  | 154 | all 64-bits when accessing streaming DMA: | 
|  | 155 |  | 
|  | 156 | int using_dac; | 
|  | 157 |  | 
|  | 158 | if (!pci_set_dma_mask(pdev, DMA_64BIT_MASK)) { | 
|  | 159 | using_dac = 1; | 
|  | 160 | } else if (!pci_set_dma_mask(pdev, DMA_32BIT_MASK)) { | 
|  | 161 | using_dac = 0; | 
|  | 162 | } else { | 
|  | 163 | printk(KERN_WARNING | 
|  | 164 | "mydev: No suitable DMA available.\n"); | 
|  | 165 | goto ignore_this_device; | 
|  | 166 | } | 
|  | 167 |  | 
|  | 168 | If a card is capable of using 64-bit consistent allocations as well, | 
|  | 169 | the case would look like this: | 
|  | 170 |  | 
|  | 171 | int using_dac, consistent_using_dac; | 
|  | 172 |  | 
|  | 173 | if (!pci_set_dma_mask(pdev, DMA_64BIT_MASK)) { | 
|  | 174 | using_dac = 1; | 
|  | 175 | consistent_using_dac = 1; | 
|  | 176 | pci_set_consistent_dma_mask(pdev, DMA_64BIT_MASK); | 
|  | 177 | } else if (!pci_set_dma_mask(pdev, DMA_32BIT_MASK)) { | 
|  | 178 | using_dac = 0; | 
|  | 179 | consistent_using_dac = 0; | 
|  | 180 | pci_set_consistent_dma_mask(pdev, DMA_32BIT_MASK); | 
|  | 181 | } else { | 
|  | 182 | printk(KERN_WARNING | 
|  | 183 | "mydev: No suitable DMA available.\n"); | 
|  | 184 | goto ignore_this_device; | 
|  | 185 | } | 
|  | 186 |  | 
|  | 187 | pci_set_consistent_dma_mask() will always be able to set the same or a | 
|  | 188 | smaller mask as pci_set_dma_mask(). However for the rare case that a | 
|  | 189 | device driver only uses consistent allocations, one would have to | 
|  | 190 | check the return value from pci_set_consistent_dma_mask(). | 
|  | 191 |  | 
| Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 192 | Finally, if your device can only drive the low 24-bits of | 
|  | 193 | address during PCI bus mastering you might do something like: | 
|  | 194 |  | 
| Tobias Klauser | 56b146d | 2006-04-10 22:54:21 -0700 | [diff] [blame] | 195 | if (pci_set_dma_mask(pdev, DMA_24BIT_MASK)) { | 
| Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 196 | printk(KERN_WARNING | 
|  | 197 | "mydev: 24-bit DMA addressing not available.\n"); | 
|  | 198 | goto ignore_this_device; | 
|  | 199 | } | 
|  | 200 |  | 
|  | 201 | When pci_set_dma_mask() is successful, and returns zero, the PCI layer | 
|  | 202 | saves away this mask you have provided.  The PCI layer will use this | 
|  | 203 | information later when you make DMA mappings. | 
|  | 204 |  | 
|  | 205 | There is a case which we are aware of at this time, which is worth | 
|  | 206 | mentioning in this documentation.  If your device supports multiple | 
|  | 207 | functions (for example a sound card provides playback and record | 
|  | 208 | functions) and the various different functions have _different_ | 
|  | 209 | DMA addressing limitations, you may wish to probe each mask and | 
|  | 210 | only provide the functionality which the machine can handle.  It | 
| Tobias Klauser | 56b146d | 2006-04-10 22:54:21 -0700 | [diff] [blame] | 211 | is important that the last call to pci_set_dma_mask() be for the | 
| Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 212 | most specific mask. | 
|  | 213 |  | 
|  | 214 | Here is pseudo-code showing how this might be done: | 
|  | 215 |  | 
|  | 216 | #define PLAYBACK_ADDRESS_BITS	DMA_32BIT_MASK | 
|  | 217 | #define RECORD_ADDRESS_BITS	0x00ffffff | 
|  | 218 |  | 
|  | 219 | struct my_sound_card *card; | 
|  | 220 | struct pci_dev *pdev; | 
|  | 221 |  | 
|  | 222 | ... | 
|  | 223 | if (!pci_set_dma_mask(pdev, PLAYBACK_ADDRESS_BITS)) { | 
|  | 224 | card->playback_enabled = 1; | 
|  | 225 | } else { | 
|  | 226 | card->playback_enabled = 0; | 
|  | 227 | printk(KERN_WARN "%s: Playback disabled due to DMA limitations.\n", | 
|  | 228 | card->name); | 
|  | 229 | } | 
|  | 230 | if (!pci_set_dma_mask(pdev, RECORD_ADDRESS_BITS)) { | 
|  | 231 | card->record_enabled = 1; | 
|  | 232 | } else { | 
|  | 233 | card->record_enabled = 0; | 
|  | 234 | printk(KERN_WARN "%s: Record disabled due to DMA limitations.\n", | 
|  | 235 | card->name); | 
|  | 236 | } | 
|  | 237 |  | 
|  | 238 | A sound card was used as an example here because this genre of PCI | 
|  | 239 | devices seems to be littered with ISA chips given a PCI front end, | 
|  | 240 | and thus retaining the 16MB DMA addressing limitations of ISA. | 
|  | 241 |  | 
|  | 242 | Types of DMA mappings | 
|  | 243 |  | 
|  | 244 | There are two types of DMA mappings: | 
|  | 245 |  | 
|  | 246 | - Consistent DMA mappings which are usually mapped at driver | 
|  | 247 | initialization, unmapped at the end and for which the hardware should | 
|  | 248 | guarantee that the device and the CPU can access the data | 
|  | 249 | in parallel and will see updates made by each other without any | 
|  | 250 | explicit software flushing. | 
|  | 251 |  | 
|  | 252 | Think of "consistent" as "synchronous" or "coherent". | 
|  | 253 |  | 
|  | 254 | The current default is to return consistent memory in the low 32 | 
|  | 255 | bits of the PCI bus space.  However, for future compatibility you | 
|  | 256 | should set the consistent mask even if this default is fine for your | 
|  | 257 | driver. | 
|  | 258 |  | 
|  | 259 | Good examples of what to use consistent mappings for are: | 
|  | 260 |  | 
|  | 261 | - Network card DMA ring descriptors. | 
|  | 262 | - SCSI adapter mailbox command data structures. | 
|  | 263 | - Device firmware microcode executed out of | 
|  | 264 | main memory. | 
|  | 265 |  | 
|  | 266 | The invariant these examples all require is that any CPU store | 
|  | 267 | to memory is immediately visible to the device, and vice | 
|  | 268 | versa.  Consistent mappings guarantee this. | 
|  | 269 |  | 
|  | 270 | IMPORTANT: Consistent DMA memory does not preclude the usage of | 
|  | 271 | proper memory barriers.  The CPU may reorder stores to | 
|  | 272 | consistent memory just as it may normal memory.  Example: | 
|  | 273 | if it is important for the device to see the first word | 
|  | 274 | of a descriptor updated before the second, you must do | 
|  | 275 | something like: | 
|  | 276 |  | 
|  | 277 | desc->word0 = address; | 
|  | 278 | wmb(); | 
|  | 279 | desc->word1 = DESC_VALID; | 
|  | 280 |  | 
|  | 281 | in order to get correct behavior on all platforms. | 
|  | 282 |  | 
| David Brownell | 21440d3 | 2006-04-01 10:21:52 -0800 | [diff] [blame] | 283 | Also, on some platforms your driver may need to flush CPU write | 
|  | 284 | buffers in much the same way as it needs to flush write buffers | 
|  | 285 | found in PCI bridges (such as by reading a register's value | 
|  | 286 | after writing it). | 
|  | 287 |  | 
| Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 288 | - Streaming DMA mappings which are usually mapped for one DMA transfer, | 
|  | 289 | unmapped right after it (unless you use pci_dma_sync_* below) and for which | 
|  | 290 | hardware can optimize for sequential accesses. | 
|  | 291 |  | 
|  | 292 | This of "streaming" as "asynchronous" or "outside the coherency | 
|  | 293 | domain". | 
|  | 294 |  | 
|  | 295 | Good examples of what to use streaming mappings for are: | 
|  | 296 |  | 
|  | 297 | - Networking buffers transmitted/received by a device. | 
|  | 298 | - Filesystem buffers written/read by a SCSI device. | 
|  | 299 |  | 
|  | 300 | The interfaces for using this type of mapping were designed in | 
|  | 301 | such a way that an implementation can make whatever performance | 
|  | 302 | optimizations the hardware allows.  To this end, when using | 
|  | 303 | such mappings you must be explicit about what you want to happen. | 
|  | 304 |  | 
|  | 305 | Neither type of DMA mapping has alignment restrictions that come | 
|  | 306 | from PCI, although some devices may have such restrictions. | 
| David Brownell | 21440d3 | 2006-04-01 10:21:52 -0800 | [diff] [blame] | 307 | Also, systems with caches that aren't DMA-coherent will work better | 
|  | 308 | when the underlying buffers don't share cache lines with other data. | 
|  | 309 |  | 
| Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 310 |  | 
|  | 311 | Using Consistent DMA mappings. | 
|  | 312 |  | 
|  | 313 | To allocate and map large (PAGE_SIZE or so) consistent DMA regions, | 
|  | 314 | you should do: | 
|  | 315 |  | 
|  | 316 | dma_addr_t dma_handle; | 
|  | 317 |  | 
|  | 318 | cpu_addr = pci_alloc_consistent(dev, size, &dma_handle); | 
|  | 319 |  | 
|  | 320 | where dev is a struct pci_dev *. You should pass NULL for PCI like buses | 
|  | 321 | where devices don't have struct pci_dev (like ISA, EISA).  This may be | 
|  | 322 | called in interrupt context. | 
|  | 323 |  | 
|  | 324 | This argument is needed because the DMA translations may be bus | 
|  | 325 | specific (and often is private to the bus which the device is attached | 
|  | 326 | to). | 
|  | 327 |  | 
|  | 328 | Size is the length of the region you want to allocate, in bytes. | 
|  | 329 |  | 
|  | 330 | This routine will allocate RAM for that region, so it acts similarly to | 
|  | 331 | __get_free_pages (but takes size instead of a page order).  If your | 
|  | 332 | driver needs regions sized smaller than a page, you may prefer using | 
|  | 333 | the pci_pool interface, described below. | 
|  | 334 |  | 
|  | 335 | The consistent DMA mapping interfaces, for non-NULL dev, will by | 
|  | 336 | default return a DMA address which is SAC (Single Address Cycle) | 
|  | 337 | addressable.  Even if the device indicates (via PCI dma mask) that it | 
|  | 338 | may address the upper 32-bits and thus perform DAC cycles, consistent | 
|  | 339 | allocation will only return > 32-bit PCI addresses for DMA if the | 
|  | 340 | consistent dma mask has been explicitly changed via | 
|  | 341 | pci_set_consistent_dma_mask().  This is true of the pci_pool interface | 
|  | 342 | as well. | 
|  | 343 |  | 
|  | 344 | pci_alloc_consistent returns two values: the virtual address which you | 
|  | 345 | can use to access it from the CPU and dma_handle which you pass to the | 
|  | 346 | card. | 
|  | 347 |  | 
|  | 348 | The cpu return address and the DMA bus master address are both | 
|  | 349 | guaranteed to be aligned to the smallest PAGE_SIZE order which | 
|  | 350 | is greater than or equal to the requested size.  This invariant | 
|  | 351 | exists (for example) to guarantee that if you allocate a chunk | 
|  | 352 | which is smaller than or equal to 64 kilobytes, the extent of the | 
|  | 353 | buffer you receive will not cross a 64K boundary. | 
|  | 354 |  | 
|  | 355 | To unmap and free such a DMA region, you call: | 
|  | 356 |  | 
|  | 357 | pci_free_consistent(dev, size, cpu_addr, dma_handle); | 
|  | 358 |  | 
|  | 359 | where dev, size are the same as in the above call and cpu_addr and | 
|  | 360 | dma_handle are the values pci_alloc_consistent returned to you. | 
|  | 361 | This function may not be called in interrupt context. | 
|  | 362 |  | 
|  | 363 | If your driver needs lots of smaller memory regions, you can write | 
|  | 364 | custom code to subdivide pages returned by pci_alloc_consistent, | 
|  | 365 | or you can use the pci_pool API to do that.  A pci_pool is like | 
|  | 366 | a kmem_cache, but it uses pci_alloc_consistent not __get_free_pages. | 
|  | 367 | Also, it understands common hardware constraints for alignment, | 
|  | 368 | like queue heads needing to be aligned on N byte boundaries. | 
|  | 369 |  | 
|  | 370 | Create a pci_pool like this: | 
|  | 371 |  | 
|  | 372 | struct pci_pool *pool; | 
|  | 373 |  | 
|  | 374 | pool = pci_pool_create(name, dev, size, align, alloc); | 
|  | 375 |  | 
|  | 376 | The "name" is for diagnostics (like a kmem_cache name); dev and size | 
|  | 377 | are as above.  The device's hardware alignment requirement for this | 
|  | 378 | type of data is "align" (which is expressed in bytes, and must be a | 
|  | 379 | power of two).  If your device has no boundary crossing restrictions, | 
|  | 380 | pass 0 for alloc; passing 4096 says memory allocated from this pool | 
|  | 381 | must not cross 4KByte boundaries (but at that time it may be better to | 
|  | 382 | go for pci_alloc_consistent directly instead). | 
|  | 383 |  | 
|  | 384 | Allocate memory from a pci pool like this: | 
|  | 385 |  | 
|  | 386 | cpu_addr = pci_pool_alloc(pool, flags, &dma_handle); | 
|  | 387 |  | 
|  | 388 | flags are SLAB_KERNEL if blocking is permitted (not in_interrupt nor | 
|  | 389 | holding SMP locks), SLAB_ATOMIC otherwise.  Like pci_alloc_consistent, | 
|  | 390 | this returns two values, cpu_addr and dma_handle. | 
|  | 391 |  | 
|  | 392 | Free memory that was allocated from a pci_pool like this: | 
|  | 393 |  | 
|  | 394 | pci_pool_free(pool, cpu_addr, dma_handle); | 
|  | 395 |  | 
|  | 396 | where pool is what you passed to pci_pool_alloc, and cpu_addr and | 
|  | 397 | dma_handle are the values pci_pool_alloc returned. This function | 
|  | 398 | may be called in interrupt context. | 
|  | 399 |  | 
|  | 400 | Destroy a pci_pool by calling: | 
|  | 401 |  | 
|  | 402 | pci_pool_destroy(pool); | 
|  | 403 |  | 
|  | 404 | Make sure you've called pci_pool_free for all memory allocated | 
|  | 405 | from a pool before you destroy the pool. This function may not | 
|  | 406 | be called in interrupt context. | 
|  | 407 |  | 
|  | 408 | DMA Direction | 
|  | 409 |  | 
|  | 410 | The interfaces described in subsequent portions of this document | 
|  | 411 | take a DMA direction argument, which is an integer and takes on | 
|  | 412 | one of the following values: | 
|  | 413 |  | 
|  | 414 | PCI_DMA_BIDIRECTIONAL | 
|  | 415 | PCI_DMA_TODEVICE | 
|  | 416 | PCI_DMA_FROMDEVICE | 
|  | 417 | PCI_DMA_NONE | 
|  | 418 |  | 
|  | 419 | One should provide the exact DMA direction if you know it. | 
|  | 420 |  | 
|  | 421 | PCI_DMA_TODEVICE means "from main memory to the PCI device" | 
|  | 422 | PCI_DMA_FROMDEVICE means "from the PCI device to main memory" | 
|  | 423 | It is the direction in which the data moves during the DMA | 
|  | 424 | transfer. | 
|  | 425 |  | 
|  | 426 | You are _strongly_ encouraged to specify this as precisely | 
|  | 427 | as you possibly can. | 
|  | 428 |  | 
|  | 429 | If you absolutely cannot know the direction of the DMA transfer, | 
|  | 430 | specify PCI_DMA_BIDIRECTIONAL.  It means that the DMA can go in | 
|  | 431 | either direction.  The platform guarantees that you may legally | 
|  | 432 | specify this, and that it will work, but this may be at the | 
|  | 433 | cost of performance for example. | 
|  | 434 |  | 
|  | 435 | The value PCI_DMA_NONE is to be used for debugging.  One can | 
|  | 436 | hold this in a data structure before you come to know the | 
|  | 437 | precise direction, and this will help catch cases where your | 
|  | 438 | direction tracking logic has failed to set things up properly. | 
|  | 439 |  | 
|  | 440 | Another advantage of specifying this value precisely (outside of | 
|  | 441 | potential platform-specific optimizations of such) is for debugging. | 
|  | 442 | Some platforms actually have a write permission boolean which DMA | 
|  | 443 | mappings can be marked with, much like page protections in the user | 
|  | 444 | program address space.  Such platforms can and do report errors in the | 
|  | 445 | kernel logs when the PCI controller hardware detects violation of the | 
|  | 446 | permission setting. | 
|  | 447 |  | 
|  | 448 | Only streaming mappings specify a direction, consistent mappings | 
|  | 449 | implicitly have a direction attribute setting of | 
|  | 450 | PCI_DMA_BIDIRECTIONAL. | 
|  | 451 |  | 
|  | be7db05 | 2005-04-17 15:26:13 -0500 | [diff] [blame] | 452 | The SCSI subsystem tells you the direction to use in the | 
|  | 453 | 'sc_data_direction' member of the SCSI command your driver is | 
|  | 454 | working on. | 
| Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 455 |  | 
|  | 456 | For Networking drivers, it's a rather simple affair.  For transmit | 
|  | 457 | packets, map/unmap them with the PCI_DMA_TODEVICE direction | 
|  | 458 | specifier.  For receive packets, just the opposite, map/unmap them | 
|  | 459 | with the PCI_DMA_FROMDEVICE direction specifier. | 
|  | 460 |  | 
|  | 461 | Using Streaming DMA mappings | 
|  | 462 |  | 
|  | 463 | The streaming DMA mapping routines can be called from interrupt | 
|  | 464 | context.  There are two versions of each map/unmap, one which will | 
|  | 465 | map/unmap a single memory region, and one which will map/unmap a | 
|  | 466 | scatterlist. | 
|  | 467 |  | 
|  | 468 | To map a single region, you do: | 
|  | 469 |  | 
|  | 470 | struct pci_dev *pdev = mydev->pdev; | 
|  | 471 | dma_addr_t dma_handle; | 
|  | 472 | void *addr = buffer->ptr; | 
|  | 473 | size_t size = buffer->len; | 
|  | 474 |  | 
|  | 475 | dma_handle = pci_map_single(dev, addr, size, direction); | 
|  | 476 |  | 
|  | 477 | and to unmap it: | 
|  | 478 |  | 
|  | 479 | pci_unmap_single(dev, dma_handle, size, direction); | 
|  | 480 |  | 
|  | 481 | You should call pci_unmap_single when the DMA activity is finished, e.g. | 
|  | 482 | from the interrupt which told you that the DMA transfer is done. | 
|  | 483 |  | 
|  | 484 | Using cpu pointers like this for single mappings has a disadvantage, | 
|  | 485 | you cannot reference HIGHMEM memory in this way.  Thus, there is a | 
|  | 486 | map/unmap interface pair akin to pci_{map,unmap}_single.  These | 
|  | 487 | interfaces deal with page/offset pairs instead of cpu pointers. | 
|  | 488 | Specifically: | 
|  | 489 |  | 
|  | 490 | struct pci_dev *pdev = mydev->pdev; | 
|  | 491 | dma_addr_t dma_handle; | 
|  | 492 | struct page *page = buffer->page; | 
|  | 493 | unsigned long offset = buffer->offset; | 
|  | 494 | size_t size = buffer->len; | 
|  | 495 |  | 
|  | 496 | dma_handle = pci_map_page(dev, page, offset, size, direction); | 
|  | 497 |  | 
|  | 498 | ... | 
|  | 499 |  | 
|  | 500 | pci_unmap_page(dev, dma_handle, size, direction); | 
|  | 501 |  | 
|  | 502 | Here, "offset" means byte offset within the given page. | 
|  | 503 |  | 
|  | 504 | With scatterlists, you map a region gathered from several regions by: | 
|  | 505 |  | 
|  | 506 | int i, count = pci_map_sg(dev, sglist, nents, direction); | 
|  | 507 | struct scatterlist *sg; | 
|  | 508 |  | 
| saeed bishara | 4c2f6d4 | 2007-08-08 13:09:00 +0200 | [diff] [blame] | 509 | for_each_sg(sglist, sg, count, i) { | 
| Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 510 | hw_address[i] = sg_dma_address(sg); | 
|  | 511 | hw_len[i] = sg_dma_len(sg); | 
|  | 512 | } | 
|  | 513 |  | 
|  | 514 | where nents is the number of entries in the sglist. | 
|  | 515 |  | 
|  | 516 | The implementation is free to merge several consecutive sglist entries | 
|  | 517 | into one (e.g. if DMA mapping is done with PAGE_SIZE granularity, any | 
|  | 518 | consecutive sglist entries can be merged into one provided the first one | 
|  | 519 | ends and the second one starts on a page boundary - in fact this is a huge | 
|  | 520 | advantage for cards which either cannot do scatter-gather or have very | 
|  | 521 | limited number of scatter-gather entries) and returns the actual number | 
|  | 522 | of sg entries it mapped them to. On failure 0 is returned. | 
|  | 523 |  | 
|  | 524 | Then you should loop count times (note: this can be less than nents times) | 
|  | 525 | and use sg_dma_address() and sg_dma_len() macros where you previously | 
|  | 526 | accessed sg->address and sg->length as shown above. | 
|  | 527 |  | 
|  | 528 | To unmap a scatterlist, just call: | 
|  | 529 |  | 
|  | 530 | pci_unmap_sg(dev, sglist, nents, direction); | 
|  | 531 |  | 
|  | 532 | Again, make sure DMA activity has already finished. | 
|  | 533 |  | 
|  | 534 | PLEASE NOTE:  The 'nents' argument to the pci_unmap_sg call must be | 
|  | 535 | the _same_ one you passed into the pci_map_sg call, | 
|  | 536 | it should _NOT_ be the 'count' value _returned_ from the | 
|  | 537 | pci_map_sg call. | 
|  | 538 |  | 
|  | 539 | Every pci_map_{single,sg} call should have its pci_unmap_{single,sg} | 
|  | 540 | counterpart, because the bus address space is a shared resource (although | 
|  | 541 | in some ports the mapping is per each BUS so less devices contend for the | 
|  | 542 | same bus address space) and you could render the machine unusable by eating | 
|  | 543 | all bus addresses. | 
|  | 544 |  | 
|  | 545 | If you need to use the same streaming DMA region multiple times and touch | 
|  | 546 | the data in between the DMA transfers, the buffer needs to be synced | 
|  | 547 | properly in order for the cpu and device to see the most uptodate and | 
|  | 548 | correct copy of the DMA buffer. | 
|  | 549 |  | 
|  | 550 | So, firstly, just map it with pci_map_{single,sg}, and after each DMA | 
|  | 551 | transfer call either: | 
|  | 552 |  | 
|  | 553 | pci_dma_sync_single_for_cpu(dev, dma_handle, size, direction); | 
|  | 554 |  | 
|  | 555 | or: | 
|  | 556 |  | 
|  | 557 | pci_dma_sync_sg_for_cpu(dev, sglist, nents, direction); | 
|  | 558 |  | 
|  | 559 | as appropriate. | 
|  | 560 |  | 
|  | 561 | Then, if you wish to let the device get at the DMA area again, | 
|  | 562 | finish accessing the data with the cpu, and then before actually | 
|  | 563 | giving the buffer to the hardware call either: | 
|  | 564 |  | 
|  | 565 | pci_dma_sync_single_for_device(dev, dma_handle, size, direction); | 
|  | 566 |  | 
|  | 567 | or: | 
|  | 568 |  | 
|  | 569 | pci_dma_sync_sg_for_device(dev, sglist, nents, direction); | 
|  | 570 |  | 
|  | 571 | as appropriate. | 
|  | 572 |  | 
|  | 573 | After the last DMA transfer call one of the DMA unmap routines | 
|  | 574 | pci_unmap_{single,sg}. If you don't touch the data from the first pci_map_* | 
|  | 575 | call till pci_unmap_*, then you don't have to call the pci_dma_sync_* | 
|  | 576 | routines at all. | 
|  | 577 |  | 
|  | 578 | Here is pseudo code which shows a situation in which you would need | 
|  | 579 | to use the pci_dma_sync_*() interfaces. | 
|  | 580 |  | 
|  | 581 | my_card_setup_receive_buffer(struct my_card *cp, char *buffer, int len) | 
|  | 582 | { | 
|  | 583 | dma_addr_t mapping; | 
|  | 584 |  | 
|  | 585 | mapping = pci_map_single(cp->pdev, buffer, len, PCI_DMA_FROMDEVICE); | 
|  | 586 |  | 
|  | 587 | cp->rx_buf = buffer; | 
|  | 588 | cp->rx_len = len; | 
|  | 589 | cp->rx_dma = mapping; | 
|  | 590 |  | 
|  | 591 | give_rx_buf_to_card(cp); | 
|  | 592 | } | 
|  | 593 |  | 
|  | 594 | ... | 
|  | 595 |  | 
|  | 596 | my_card_interrupt_handler(int irq, void *devid, struct pt_regs *regs) | 
|  | 597 | { | 
|  | 598 | struct my_card *cp = devid; | 
|  | 599 |  | 
|  | 600 | ... | 
|  | 601 | if (read_card_status(cp) == RX_BUF_TRANSFERRED) { | 
|  | 602 | struct my_card_header *hp; | 
|  | 603 |  | 
|  | 604 | /* Examine the header to see if we wish | 
|  | 605 | * to accept the data.  But synchronize | 
|  | 606 | * the DMA transfer with the CPU first | 
|  | 607 | * so that we see updated contents. | 
|  | 608 | */ | 
|  | 609 | pci_dma_sync_single_for_cpu(cp->pdev, cp->rx_dma, | 
|  | 610 | cp->rx_len, | 
|  | 611 | PCI_DMA_FROMDEVICE); | 
|  | 612 |  | 
|  | 613 | /* Now it is safe to examine the buffer. */ | 
|  | 614 | hp = (struct my_card_header *) cp->rx_buf; | 
|  | 615 | if (header_is_ok(hp)) { | 
|  | 616 | pci_unmap_single(cp->pdev, cp->rx_dma, cp->rx_len, | 
|  | 617 | PCI_DMA_FROMDEVICE); | 
|  | 618 | pass_to_upper_layers(cp->rx_buf); | 
|  | 619 | make_and_setup_new_rx_buf(cp); | 
|  | 620 | } else { | 
|  | 621 | /* Just sync the buffer and give it back | 
|  | 622 | * to the card. | 
|  | 623 | */ | 
|  | 624 | pci_dma_sync_single_for_device(cp->pdev, | 
|  | 625 | cp->rx_dma, | 
|  | 626 | cp->rx_len, | 
|  | 627 | PCI_DMA_FROMDEVICE); | 
|  | 628 | give_rx_buf_to_card(cp); | 
|  | 629 | } | 
|  | 630 | } | 
|  | 631 | } | 
|  | 632 |  | 
|  | 633 | Drivers converted fully to this interface should not use virt_to_bus any | 
|  | 634 | longer, nor should they use bus_to_virt. Some drivers have to be changed a | 
|  | 635 | little bit, because there is no longer an equivalent to bus_to_virt in the | 
|  | 636 | dynamic DMA mapping scheme - you have to always store the DMA addresses | 
|  | 637 | returned by the pci_alloc_consistent, pci_pool_alloc, and pci_map_single | 
|  | 638 | calls (pci_map_sg stores them in the scatterlist itself if the platform | 
|  | 639 | supports dynamic DMA mapping in hardware) in your driver structures and/or | 
|  | 640 | in the card registers. | 
|  | 641 |  | 
|  | 642 | All PCI drivers should be using these interfaces with no exceptions. | 
|  | 643 | It is planned to completely remove virt_to_bus() and bus_to_virt() as | 
|  | 644 | they are entirely deprecated.  Some ports already do not provide these | 
|  | 645 | as it is impossible to correctly support them. | 
|  | 646 |  | 
| Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 647 | Optimizing Unmap State Space Consumption | 
|  | 648 |  | 
|  | 649 | On many platforms, pci_unmap_{single,page}() is simply a nop. | 
|  | 650 | Therefore, keeping track of the mapping address and length is a waste | 
|  | 651 | of space.  Instead of filling your drivers up with ifdefs and the like | 
|  | 652 | to "work around" this (which would defeat the whole purpose of a | 
|  | 653 | portable API) the following facilities are provided. | 
|  | 654 |  | 
|  | 655 | Actually, instead of describing the macros one by one, we'll | 
|  | 656 | transform some example code. | 
|  | 657 |  | 
|  | 658 | 1) Use DECLARE_PCI_UNMAP_{ADDR,LEN} in state saving structures. | 
|  | 659 | Example, before: | 
|  | 660 |  | 
|  | 661 | struct ring_state { | 
|  | 662 | struct sk_buff *skb; | 
|  | 663 | dma_addr_t mapping; | 
|  | 664 | __u32 len; | 
|  | 665 | }; | 
|  | 666 |  | 
|  | 667 | after: | 
|  | 668 |  | 
|  | 669 | struct ring_state { | 
|  | 670 | struct sk_buff *skb; | 
|  | 671 | DECLARE_PCI_UNMAP_ADDR(mapping) | 
|  | 672 | DECLARE_PCI_UNMAP_LEN(len) | 
|  | 673 | }; | 
|  | 674 |  | 
|  | 675 | NOTE: DO NOT put a semicolon at the end of the DECLARE_*() | 
|  | 676 | macro. | 
|  | 677 |  | 
|  | 678 | 2) Use pci_unmap_{addr,len}_set to set these values. | 
|  | 679 | Example, before: | 
|  | 680 |  | 
|  | 681 | ringp->mapping = FOO; | 
|  | 682 | ringp->len = BAR; | 
|  | 683 |  | 
|  | 684 | after: | 
|  | 685 |  | 
|  | 686 | pci_unmap_addr_set(ringp, mapping, FOO); | 
|  | 687 | pci_unmap_len_set(ringp, len, BAR); | 
|  | 688 |  | 
|  | 689 | 3) Use pci_unmap_{addr,len} to access these values. | 
|  | 690 | Example, before: | 
|  | 691 |  | 
|  | 692 | pci_unmap_single(pdev, ringp->mapping, ringp->len, | 
|  | 693 | PCI_DMA_FROMDEVICE); | 
|  | 694 |  | 
|  | 695 | after: | 
|  | 696 |  | 
|  | 697 | pci_unmap_single(pdev, | 
|  | 698 | pci_unmap_addr(ringp, mapping), | 
|  | 699 | pci_unmap_len(ringp, len), | 
|  | 700 | PCI_DMA_FROMDEVICE); | 
|  | 701 |  | 
|  | 702 | It really should be self-explanatory.  We treat the ADDR and LEN | 
|  | 703 | separately, because it is possible for an implementation to only | 
|  | 704 | need the address in order to perform the unmap operation. | 
|  | 705 |  | 
|  | 706 | Platform Issues | 
|  | 707 |  | 
|  | 708 | If you are just writing drivers for Linux and do not maintain | 
|  | 709 | an architecture port for the kernel, you can safely skip down | 
|  | 710 | to "Closing". | 
|  | 711 |  | 
|  | 712 | 1) Struct scatterlist requirements. | 
|  | 713 |  | 
|  | 714 | Struct scatterlist must contain, at a minimum, the following | 
|  | 715 | members: | 
|  | 716 |  | 
|  | 717 | struct page *page; | 
|  | 718 | unsigned int offset; | 
|  | 719 | unsigned int length; | 
|  | 720 |  | 
|  | 721 | The base address is specified by a "page+offset" pair. | 
|  | 722 |  | 
|  | 723 | Previous versions of struct scatterlist contained a "void *address" | 
|  | 724 | field that was sometimes used instead of page+offset.  As of Linux | 
|  | 725 | 2.5., page+offset is always used, and the "address" field has been | 
|  | 726 | deleted. | 
|  | 727 |  | 
|  | 728 | 2) More to come... | 
|  | 729 |  | 
|  | 730 | Handling Errors | 
|  | 731 |  | 
|  | 732 | DMA address space is limited on some architectures and an allocation | 
|  | 733 | failure can be determined by: | 
|  | 734 |  | 
|  | 735 | - checking if pci_alloc_consistent returns NULL or pci_map_sg returns 0 | 
|  | 736 |  | 
|  | 737 | - checking the returned dma_addr_t of pci_map_single and pci_map_page | 
|  | 738 | by using pci_dma_mapping_error(): | 
|  | 739 |  | 
|  | 740 | dma_addr_t dma_handle; | 
|  | 741 |  | 
|  | 742 | dma_handle = pci_map_single(dev, addr, size, direction); | 
|  | 743 | if (pci_dma_mapping_error(dma_handle)) { | 
|  | 744 | /* | 
|  | 745 | * reduce current DMA mapping usage, | 
|  | 746 | * delay and try again later or | 
|  | 747 | * reset driver. | 
|  | 748 | */ | 
|  | 749 | } | 
|  | 750 |  | 
|  | 751 | Closing | 
|  | 752 |  | 
|  | 753 | This document, and the API itself, would not be in it's current | 
|  | 754 | form without the feedback and suggestions from numerous individuals. | 
|  | 755 | We would like to specifically mention, in no particular order, the | 
|  | 756 | following people: | 
|  | 757 |  | 
|  | 758 | Russell King <rmk@arm.linux.org.uk> | 
|  | 759 | Leo Dagum <dagum@barrel.engr.sgi.com> | 
|  | 760 | Ralf Baechle <ralf@oss.sgi.com> | 
|  | 761 | Grant Grundler <grundler@cup.hp.com> | 
|  | 762 | Jay Estabrook <Jay.Estabrook@compaq.com> | 
|  | 763 | Thomas Sailer <sailer@ife.ee.ethz.ch> | 
|  | 764 | Andrea Arcangeli <andrea@suse.de> | 
| Rob Landley | 26bbb29 | 2007-10-15 11:42:52 +0200 | [diff] [blame] | 765 | Jens Axboe <jens.axboe@oracle.com> | 
| Linus Torvalds | 1da177e | 2005-04-16 15:20:36 -0700 | [diff] [blame] | 766 | David Mosberger-Tang <davidm@hpl.hp.com> |