| Bryan Schumaker | 114a0a0 | 2011-11-01 13:35:23 -0400 | [diff] [blame] | 1 | |
| 2 | Fault Injection |
| 3 | =============== |
| 4 | Fault injection is a method for forcing errors that may not normally occur, or |
| 5 | may be difficult to reproduce. Forcing these errors in a controlled environment |
| 6 | can help the developer find and fix bugs before their code is shipped in a |
| 7 | production system. Injecting an error on the Linux NFS server will allow us to |
| 8 | observe how the client reacts and if it manages to recover its state correctly. |
| 9 | |
| 10 | NFSD_FAULT_INJECTION must be selected when configuring the kernel to use this |
| 11 | feature. |
| 12 | |
| 13 | |
| 14 | Using Fault Injection |
| 15 | ===================== |
| 16 | On the client, mount the fault injection server through NFS v4.0+ and do some |
| 17 | work over NFS (open files, take locks, ...). |
| 18 | |
| 19 | On the server, mount the debugfs filesystem to <debug_dir> and ls |
| 20 | <debug_dir>/nfsd. This will show a list of files that will be used for |
| 21 | injecting faults on the NFS server. As root, write a number n to the file |
| 22 | corresponding to the action you want the server to take. The server will then |
| 23 | process the first n items it finds. So if you want to forget 5 locks, echo '5' |
| 24 | to <debug_dir>/nfsd/forget_locks. A value of 0 will tell the server to forget |
| 25 | all corresponding items. A log message will be created containing the number |
| 26 | of items forgotten (check dmesg). |
| 27 | |
| 28 | Go back to work on the client and check if the client recovered from the error |
| 29 | correctly. |
| 30 | |
| 31 | |
| 32 | Available Faults |
| 33 | ================ |
| 34 | forget_clients: |
| 35 | The NFS server keeps a list of clients that have placed a mount call. If |
| 36 | this list is cleared, the server will have no knowledge of who the client |
| 37 | is, forcing the client to reauthenticate with the server. |
| 38 | |
| 39 | forget_openowners: |
| 40 | The NFS server keeps a list of what files are currently opened and who |
| 41 | they were opened by. Clearing this list will force the client to reopen |
| 42 | its files. |
| 43 | |
| 44 | forget_locks: |
| 45 | The NFS server keeps a list of what files are currently locked in the VFS. |
| 46 | Clearing this list will force the client to reclaim its locks (files are |
| 47 | unlocked through the VFS as they are cleared from this list). |
| 48 | |
| 49 | forget_delegations: |
| 50 | A delegation is used to assure the client that a file, or part of a file, |
| 51 | has not changed since the delegation was awarded. Clearing this list will |
| 52 | force the client to reaquire its delegation before accessing the file |
| 53 | again. |
| 54 | |
| 55 | recall_delegations: |
| 56 | Delegations can be recalled by the server when another client attempts to |
| 57 | access a file. This test will notify the client that its delegation has |
| 58 | been revoked, forcing the client to reaquire the delegation before using |
| 59 | the file again. |
| 60 | |
| 61 | |
| 62 | tools/nfs/inject_faults.sh script |
| 63 | ================================= |
| 64 | This script has been created to ease the fault injection process. This script |
| 65 | will detect the mounted debugfs directory and write to the files located there |
| 66 | based on the arguments passed by the user. For example, running |
| 67 | `inject_faults.sh forget_locks 1` as root will instruct the server to forget |
| 68 | one lock. Running `inject_faults forget_locks` will instruct the server to |
| 69 | forgetall locks. |