Wednesday, April 15, 2009

solaris troubleshooting

(7.1) What can I do if Solaris won't boot?
You need to boot from your install CD. Insert the Solaris Software CD in your CDROM drive. If your CDROM drive/BIOS isn't bootable, first insert the "Device Configuration Assistant" (DCA) diskette. At the "Boot Solaris" menu, choose "CD."
At the "Type of Installation: Interactive or JumpStart" menu, type "b -s"
Or, after the video configuration, network, time and date you'll notice one of the menu's has a button: [Exit] Select Exit and, when it asks you again "do you want to exit?," just say yes.
Once you're at the UNIX root prompt #, you can mount the boot drive with "mount /dev/dsk/c0t0d0s0 /mnt"" and view anything wrong with the boot drive (omit the "t0" for ATAPI).
[Modified from Bob Palowoda's Solaris 2.4 x86 FAQ]

(7.2) How do I restore the Solaris boot block without reinstalling?
This may happen when installing a boot manager that comes with another operating system (such as LILO from Linux) or an after-market multi-OS boot manager. These sometimes trample's active partition, which in our case is Solaris. Also, moving the Solaris partition with a partition manager program such as Partition Magic requires reinstalling the Solaris boot block. Before taking these steps, first verify the Solaris partition is active. If it isn't, just make the Solaris partition active and reboot. Otherwise follow the steps below.
1. Boot from CD-ROM and get the root prompt, #, as described in the previous question, 7.1.
2. Determine the controller, disk number, and partition. The boot disk is /dev/rdsk/c?t?d?p? where ? is the controller #, target ID, and disk #, and partition #. Omit "t?" for ATAPI E.g., /dev/rdsk/c0d0p0
3. Verify it's the correct device correct with prtvtoc for the drive: This is VERY important; if it's wrong, you you may hose another partition: prtvtoc /dev/rdsk/c0t0d0p0 (omit "t0" for ATAPI, always use p0, which means the "entire drive"). The prtvtoc prints out the map for the Solaris partition on the hard drive, if found. The partitions shown on the output are actually "slices" within the Solaris partition.
4. Restore the boot block as follows:
/sbin/fdisk -b /usr/lib/fs/ufs/mboot (raw disk dev)
E.g., for SCSI it might be:
/sbin/fdisk -b /usr/lib/fs/ufs/mboot /dev/rdsk/c0t0d0p0
(omit "t0" for ATAPI)
5. Finally, remove your CDROM and diskette media and type "/sbin/shutdown -i6" to reboot. The Solaris Multiple Device Boot Menu should appear after rebooting. If not, you can always to an upgrade (re-)install.
Note: This procedure does NOT make your Solaris partition active again (sometimes needed after installing another operating system, such as Windows, on the same disk), it just writes to your bootblock IN your Solaris partition. To learn more about the Solaris boot process, read the boot(1M) man page.

(7.3) What can I do during the Solaris/x86 booting sequence?
Starting with Solaris 10 06/06, Solaris now uses GRUB to boot, making the answer below obsolete. That means you can boot to multiple partitions, with multiple instances of Solaris and other operating systems (such as Linux and Windows)
Step #1: Boot loader
If you have multiple partitions, the boot loader in the Solaris partition will come up and ask you which partition you want to boot. This partition must be the active partition, or at least be marked active by a third-party boot manager before this boot loader receives control (not all boot managers have this feature). If you don't answer in so many seconds, it boots Solaris.
This boot manager is pretty basic. It has no customization. You can't change the default boot partition to one other than Solaris, you can't change the timeout value, and you can't change the partition descriptions. But it gets the job done.
Step #2: Device Configuration Assistant (DCA)
This will ask you to press ESC if you want to change stuff. This is to make up for the fact that x86 machines don't have a nice OpenBOOT chip to sort out REAL "Plug and Play".
Basically, in Solaris x86, the Device Assistant seems to set up certain things in /boot/solaris. This is so the "real" OS has some common format to examine for devices, instead of having lots of nasty x86 hardware specific stuff. That way, Sun can keep the main OS somewhat hardware independent, and keep it very close to the Sparc version.
The "Assistant" can actually been of assistance. If you select "partial scan", then "Device tasks", and then "View/Edit Devices", it will tell you what Solaris THINKS your devices are, and where they are at. Quite useful, when Solaris gets completely lost, and you're wondering if it's your fault, or what.
Otherwise, it can give you a warm fuzzy feeling, if you select "Full Scan", and you see all your devices properly recognized.
Step #3: OS Boot
Well, actually, the "Boot Assistant". The interface is similar, but not identical, to SPARC Solaris' OpenBoot 'boot' command. The main differences I notice are:
• It's "b -r", not "boot -r", if you want to force reconfiguration. Why would you want to do that? Well sometimes, if your devices have changed a LITTLE, you might want to do this. You can also invoke it with a "touch /reconfigure" as root before rebooting with "/sbin/shutdown -i6".
• You don't have nice device aliases like you do with Sparc hardware. That being said, if you do nothing, it should autoboot into the actual OS in a few seconds. Or you can type something quickly within 5 seconds, and force a "reconfiguration boot", as mentioned earlier.
• The "Driver Assistant" or whatever, really looks for major changes, like adding or removing a card. However, if, say, you add another disk drive, you'll probably want to just do "b -r". Note that the Device Assistant will itself trigger a "b -r", after it has autodetected hardware changes.
Step #4: The Main OS: Solaris
You made it (I hope)!. Hopefully, you should now see a line with "SunOS5.8" or similar in it, and a little twirly text character spinner starting. You are now really in the classic Solaris environment. From here on in, your experience is almost identical to your brethren who work with SPARC Sun equipment.
To learn more about the the Solaris boot process, read the boot(1M) man page.
[Thanks to Phil at]

(7.4) How do I logon as root if the password doesn't work anymore?
Regaining control of a Solaris x86 system where the root password has been lost can be accomplished by the following steps. Note that any savvy user can do this with the proper CD-ROM and diskette. Therefore, of course, physical security of a system is important for machines containing sensitive data.
1. Insert installation boot diskette and installation CD-ROM for Solaris x86.
2. Boot system from the installation floppy and select the CD-ROM as the boot device.
3. Type "b -s" (instead of typing 1 or 2 from the menu) and it'll drop you straight to a root shell, #, (and you'll be in single-user mode).
4. At the root prompt, #, key in the following commands, which will create a directory called hdrive under the /tmp directory and then mount the root hard drive partition under this temporary directory.
5. mkdir /tmp/hdrive
6. mount /dev/dsk/c0t0d0s0 /tmp/hdrive #SCSI; for ATAPI, omit "t0"
7. To use the vi editor, the TERM variable must be defined. Key in the following commands.
8. TERM=at386
9. export TERM
10. Start vi (or some other editor) and load /tmp/hdrive/etc/shadow file:
11. vi /tmp/hdrive/etc/shadow
12. Change the first line of the shadow file that has the root entry to:
13. root::6445::::::
14. Write and quit the vi editor with the "!" override command:
15. :wq!
16. Remove the floppy installation diskette, and reboot the system:
17. /sbin/shutdown -i6
18. When system has rebooted from the hard drive, you can now log in from the Console Login: as root with no password. Just hit enter for the password.
19. After logging in as root, use the passwd command to change the root password and secure the system.
Andreas Pfaffeneder has a simpler suggested to recover the password:
Choose the Failsafe-Boot option (which results in kernel/unix -s), answer "Yes" when you are prompted if / of the installed system should be mounted. Chroot into the system and change the password:
# chroot /a /bin/bash
# passwd
# /sbin/shutdown -i6
[Thanks to Lynn R. Francis of Texas State Technical College and Andreas Pfaffeneder]

(7.5) My licensed software fails because the host ID is 0. What's wrong?
Intel processor machines don't have an IDPROM, so Sun generates a serial number, hostid command or sysinfo()'s SI_HW_SERIAL, pseudo-randomly during installation. The number is stored in /kernel/misc/sysinit, whose only function, it appears, is to provide the serial number. If serialization information is tampered or sysinit fails to load, the host ID will be 0. If you reinstall Solaris, sysinit will be regenerated and your host ID will change. So be careful about reinstalling Solaris if you have licensed software that depends on your host ID. Backup your sysinit file.
To preserve the same ID (and therefore licenses), copy file /kernel/misc/sysinit to the replacement system. I understand the Sun Workshop/Sun ONE compiler manual says this is allowed twice per calendar year (please verify this yourself).
For more information, see the Sun NVRAM/hostid FAQ, available at and elsewhere. This also has tools to fake hostids.

(7.6) How can I fix Netscape Communicator to render fonts correctly on S/x86?
This problem occurs with Solaris 2.6 and Netscape Communicator 4.0x, and has since been fixed. Apply patch 106248, which I'm told fixes this problem. A workaround is to add the following two lines to your ~/.xinitrc file:
xset +fp /usr/openwin/lib/X11/fonts/75dpi/
xset fp rehash
Another workaround, if you don't have these fonts, is to go into Netscape Preferences and change the font faces.
[Thank's to Alan Orndorff, Jeffrey Cook, and John Riddoch]

(7.7) Why doesn't Netscape run as root?
This is a bug in Netscape. Due to a Netscape 4.x bug (it thrashes the $HOME environment variable) the X11 library cannot find root's .Xauthority file in the root dir unless your current directory is /.
Large, complex programs (especially those taking input from & to the Internet) should not be run as root. Experienced users and Administrators run as root only for essential sysadmin tasks.
If you must run as root, try one of these tricks:
• Make sure you run Netscape with a current directory of / (so that the relative pathname '.Xauthority' works)
• Or start Netscape with HOME="/." (so that Netscape doesn't thrash the $HOME variable)
• Or start Netscape with an environment variable XAUTHORITY="/.Xauthority"
[Thanks to Jürgen Keil via John Groenveld]

(7.8) I moved my PCI host adapter to another slot and the system won't boot!
Don't move the adapter. It isn't a supported feature in Solaris and isn't easy to recover from. If you have any choice in the matter, move the controller back to it original slot.
The PCI device number is part of the device's basic ID, including its child disks. If you change slots, you've effectively removed that controller and its disks, and added an unrelated controller and disks. You need to fix up all of the references to the old disks to point to the new disks.
I've never come up with any strategy better than "boot, observe failure, fix failure, reboot" for recovering from this kind of change. For simple cases (single controller, in particular) it can be helpful to clear /dev/dsk/* and /dev/rdsk/* and run "disks", but that is perilous too.
Incidentally, changing motherboards is likely to trip exactly this problem, because motherboards generally number their slots differently.
To conclude, it's difficult and dangerous, and the general guidelines involves fixing:
1. /etc/vfstab or /dev or both
2. /devices to match one another
3. possibly removing lines from /etc/path_to_inst in order to make the right /devices nodes show up
The ultimate goal is to get back the same controller numbers as before.
[Sun FAQ 2576-02 at http://access1.Sun.COM/cgi-bin/rinfo2html?257602.faq]

(7.9) Why is Solaris always booting into the Device Configuration Assistant (DCA)?
This is usually caused by one of the following:
• You installed Solaris onto a disk other than the primary boot disk.
• You didn't remove your DCA boot diskette or if you didn't remove your installation CD-ROM if it's in a bootable CD-ROM drive.
• File /boot/solaris/bootenv.rc is corrupt or truncated, usually after a hard reboot or reset. This file is setup and used by DCA. It should contain several lines.
To change or set your default boot device, See Sun FAQ 2271-02 at http://access1.Sun.COM/cgi-bin/rinfo2html?227102.faq for instructions. To summarize:
• From the "Boot Solaris" screen, press F4 (Boot Tasks).
• On the "Boot Tasks" screen, press Enter to place an "X" in front of "View/Edit Autoboot Settings."
• In the "View/Edit Autoboot Settings" screen, note that the Default Boot Device will not be set to any valid device. Place an "X" in front of Set Default Boot Device and press F2 (Continue).
• On the Set Default Boot Device screen, place an X in front of the correct disk and press F2 (Continue).
• Arrow up to the Accept Settings and press Enter to mark with an "X". Press F2 (Continue) to return to the Boot Tasks screen.
• Press F3 (Back). It will load appropriate drivers after which you will be at the Boot Solaris screen. Press F2 (Continue) to continue booting.

(7.10) What is the equivalent of STOP-A for Solaris x86?
>I don't think so, because Stop-A allow you to go into open boot prom of >the SUN and on a x86 it's a different thing (BIOS) Unlike Solaris on Sparc (where STOP-A gets you the OpenBoot prompt), there is no PROM firmware to drop into on x86. You can boot your system under kadb and then use a similar keystroke to drop into kadb and obtain debugging information. To boot under kadb, type eeprom boot-file=kadb and then:
• Prior to Solaris 8, type CTRL-ALT-d to jump into kadb
• On Solaris 8 or later, type F1-a to jump into kadb
• At the Solaris x86 Boot Assistant prompt type "b kadb"
You can then type, for example:
$to force your system to panic and generate a crash dump (the equivalent of "sync" at the ok prompt on SPARC).
The Device Configuration Assistant (DCA) portion of the Intel boot process can be interrupted by hitting escape when prompted. This I feel is the Intel version of the Boot Prom Monitor. Of course, all the commands cannot be equated apples to apples because of the hardware differences!
If your console is a terminal, you can type "shift-break" or "ctrl-break" or "ctrl-\" (ctrl-backslash) or "" followed by "~" and "ctrl-break" on Solaris Sparc, but this, too, is not available for Solaris x86.
With Solaris 8 SPARC (but not Intel), there's a new feature to allow keyboard sequences to generate a break (bug 4147705). The 3-character sequence is , ~ (tilde), ^b. Each character must be entered between 0.5 to 2 seconds. This is enabled with the "kbd -a alternate" command.
Similarly a soft reset is , ~ (tilde), cntl-shift-R, XIR is , ~ (tilde), cntl-shift-X, and Power Cycle is , ~ (tilde), cntl-shift-P, I believe these commands are also available only on SPARC.
[Thanks to Ramit Luthra and Mike Shapiro]

(7.11) How can I reboot Solaris x86 without it asking me to to "press a key" before rebooting?
This works for me: become root and type "shutdown -i6 -g0 -y". Or: "/sbin/shutdown -i6 This is most useful when the system is remote with no console keyboard access. don't use reboot, halt, or poweroff as they bypass the shutdown scripts.
[Thanks to Charles J. Fisher]

(7.12) Help! I'm stuck in the "Boot Assistant" and can't boot. What do I do?
If you get a message similar to: Run Error: File not found. could not run s You probably typed "reboot -- -s" or "reboot -- -r" or similar. This works for Solaris SPARC, but not for Solaris on Intel--it's disastrous. It changes your "boot-file" eeprom variable to "-s", which errors out and puts you in an endless loop in B oot Assistant.
To undo this, type the following at the Boot Assistant prompt: "b kernel/unix" This boots with file /platform/i86/kernel/unix. If this doesn't help, your filesystem may be hosed. In that case, you have to reinstall. But make sure this is the case first.

(7.13) Help! I get error 2 or error 8 while applying patches. What do I do?
Don't do anything. Error 2 means you already have the same or newer code. Error 8 means you can't patch some optional packages that haven't been installed, even if you did "everything plus OEM" during the original installation. Other errors, usually from lack of disk space, are explained in the patchadd(1M) man page.
[Thanks to Paul Karagianis]

(7.14) How do I prevent kdmconfig from running on boot up when I know my keyboard, display, and mouse configuration has not changed?
Mike of Sun has this response (9/2002):
I recognized this as a bug that was fixed a while back, for one instance with older ATI cards. I mentioned it to the video developer that fixed the ATI bug and he mentioned that there is a workaround if you see this:
This problem occurs with certain hardware (keyboards, mice, video devices). During booting, a checksum is calculated based on some info obtained for each device. The checksum is compared to a checksum recorded in the OWconfig file. If the checksums don't match, kdmconfig thinks the device may have changed, and asks the user to check it.
On systems that exhibit this problem, the device info that is checksummed seems to change from boot to boot even though no hardware has changed. I've seen this happen with some old ATI video cards and some keyboards.
The easy workaround for the problem is to run kdmconfig and test and accept the desired configuration by clicking on the "Yes" button of the test display. Then edit the last line of the OWconfig file in /etc/openwin/server/etc. Change the "1" to "2", so that is says: TestedByUser="2"; This will cause kdmconfig to ignore checksum differences.
If you are upgrading from Solaris 8 or older to Solaris 9, check the ddxHandler line. It should say "", not "". Otherwise, X Windows won't start (no graphics).
[Thanks to Mike Riley]

(7.15) I get this error message: "can't get local host's domain name" or "The local host's domain name hasn't been set." What do I do?
This is a NIS message. The easiest way to fix it is to type the following as root:
domainname; domainname >/etc/defaultdomain
(replace with your NIS domain name, which is usually the same as the DNS domain name).

(7.16) My system doesn't boot due to superblock problems with the root filesystem. What do I do?
Normally, you reboot in single user mode and run /usr/bin/fsck as root and everything is OK. If you get a message about errors/problems on /dev/dsk/c0d0s0, are told to run fsck manually in single user mode, and get this message:
then you may be able to recover from this if the disk isn't entirely corrupted. The superblock stores important information about the file system. Because it is so important it is duplicated in several places. Hopefully one of the backup superblocks isn't corrupted. To see duplicate locations of superblock, use newfs -Nv. For example, if your root slice is at /dev/dsk/c0d0s0, run this command:
# newfs -Nv /dev/dsk/c0d0s0 You must specify -Nv so you don't clobber your root slice with a new filesystem. Your output should look like this:
# newfs -Nv /dev/dsk/c0d0s0
mkfs -F ufs -o N /dev/rdsk/c0d0s0 614880 63 16 8192 1024 16 10 60 2048 t
0 -1 8
7 n
/dev/rdsk/c0d0s0: 614880 sectors in 610 cylinders of 16 tracks, 63
300.2MB in 39 cyl groups (16 c/g, 7.88MB/g, 3776 i/g)
super-block backups (for fsck -F ufs -o b=#) at:
32, 16224, 32416, 48608, 64800, 80992, 97184, 113376, 129568, 145760,
468576, 484768, 500960, 516128, 532320, 548512, 564704, 580896, 597088,
Note the numbers following "super-block backups." Use one of the numbers in fsck (e.g., 32) and use it with the fsck -F -o b= option:
# fsck -F ufs -o b=32
You may get a message FREE BLK COUNT(S) WRONG IN SUPERBLOCK SALVAGE? or FILE SYSTEM STATE IN SUPERBLOCK IS WRONG; FIX? In either case, type "yes" and press return. You should get a FILE SYSTEM WAS MODIFIED message. Reboot your system. If system complains about shutdown not being found do a halt -q. Now, hopefully, your system will boot up with out any problems.
[Thanks to Kevin Smith.]

(7.17) My system doesn't boot because the boot archive is corrupt. What do I do?
The boot archive contains the kernel modules and configuration files that are required to boot your machine. Boot up in Solaris failsafe mode. Your Solaris image should be mounted on /a. Type the following:
rm -f /a/platform/i86pc/boot_archive; bootadm update-archive -R /a

Solaris SPARC Boot Sequence

The following represents a summary of the boot process for a Solaris 2.x system on Sparc hardware.
Power On: Depending on the system involved, you may see some output on a serial terminal immediately after power on. This may take the form of a Hardware Power ON message on a large Enterprise server, or a "'" or "," in the case of an older Ultra system. These indications will not be present on a monitor connected directly to the server.
POST: If the PROM diag-switch? parameter is set to true, output from the POST (Power On Self Test) will be viewable on a serial terminal. The PROM diag-level parameter determines the extent of the POST tests. (See the Hardware Diagnostics page for more information on these settings.) If a serial terminal is not connected, a prtdiag -v will show the results of the POST once the system has booted. If a keyboard is connected, it will beep and the keyboard lights will flash during POST. If the POST fails, an error indication may be displayed following the failure.
Init System: The "Init System" process can be broken down into several discrete parts:
• OBP: If diag-switch? is set, an Entering OBP message will be seen on a serial terminal. The MMU (memory management unit) is enabled.
• NVRAM: If use-nvramrc? is set to true, read the NVRAMRC. This may contain information about boot devices, especially where the boot disk has been encapsulated with VxVM or DiskSuite.
• Probe All: This includes checking for SCSI or other disk drives and devices.
• Install Console: At this point, a directly connected monitor and keyboard will become active, or the serial port will become the system console access. If a keyboard is connected to the system, the lights will flash again during this step.
• Banner: The PROM banner will be displayed. This banner includes a logo, system type, PROM revision level, the ethernet address, and the hostid.
• Create Device Tree: The hardware device tree will be built. This device tree can be explored using PROM monitor commands at the ok> prompt, or by using prtconf once the system has been booted.
Extended Diagnostics: If diag-switch? and diag-level are set, additional diagnostics will appear on the system console.
auto-boot?: If the auto-boot? PROM parameter is set, the boot process will begin. Otherwise, the system will drop to the ok> PROM monitor prompt, or (if sunmon-compat? and security-mode are set) the > security prompt.
The boot process will use the boot-device and boot-file PROM parameters unless diag-switch? is set. In this case, the boot process will use the diag-device and diag-file.
bootblk: The OBP (Open Boot PROM) program loads the bootblk primary boot program from the boot-device (or diag-device, if diag-switch? is set). If the bootblk is not present or needs to be regenerated, it can be installed by running the installboot command after booting from a CDROM or the network. A copy of the bootblk is available at /usr/platform/`arch -k`/lib/fs/ufs/bootblk
ufsboot: The secondary boot program, /platform/`arch -k`/ufsboot is run. This program loads the kernel core image files. If this file is corrupted or missing, a bootblk: can't find the boot program or similar error message will be returned.
kernel: The kernel is loaded and run. For 32-bit Solaris systems, the relevant files are:
• /platform/`arch -k`/kernel/unix
• /kernel/genunix
For 64-bit Solaris systems, the files are:
• /platform/`arch -k`/kernel/sparcV9/unix
• /kernel/genunix
As part of the kernel loading process, the kernel banner is displayed to the screen. This includes the kernel version number (including patch level, if appropriate) and the copyright notice.
The kernel initializes itself and begins loading modules, reading the files with the ufsboot program until it has loaded enough modules to mount the root filesystem itself. At that point, ufsboot is unmapped and the kernel uses its own drivers. If the system complains about not being able to write to the root filesystem, it is stuck in this part of the boot process.
The boot -a command singlesteps through this portion of the boot process. This can be a useful diagnostic procedure if the kernel is not loading properly.
/etc/system: The /etc/system file is read by the kernel, and the system parameters are set.
The following types of customization are available in the /etc/system file:
• moddir: Changes path of kernel modules.
• forceload: Forces loading of a kernel module.
• exclude: Excludes a particular kernel module.
• rootfs: Specify the system type for the root file system. (ufs is the default.)
• rootdev: Specify the physical device path for root.
• set: Set the value of a tuneable system parameter.
If the /etc/system file is edited, it is strongly recommended that a copy of the working file be made to a well-known location. In the event that the new /etc/system file renders the system unbootable, it might be possible to bring the system up with a boot -a command that specifies the old file. If this has not been done, the system may need to be booted from CD or network so that the file can be mounted and edited.
kernel initialized: The kernel creates PID 0 ( sched). The sched process is sometimes called the "swapper."
init: The kernel starts PID 1 (init).
init: The init process reads the /etc/inittab and /etc/default/init and follows the instructions in those files.
Some of the entries in the /etc/inittab are:
• fs: sysinit (usually /etc/rcS)
• is: default init level (usually 3, sometimes 2)
• s#: script associated with a run level (usually /sbin/rc#)
rc scripts: The rc scripts execute the files in the /etc/rc#.d directories. They are run by the /sbin/rc# scripts, each of which corresponds to a run level.
Debugging can often be done on these scripts by adding echo lines to a script to print either a "I got this far" message or to print out the value of a problematic variable.

Sun CD ROM Troubleshooting
Many CD ROM problems are software rather than hardware problems. If your problem is hardware-related, you can look at our Hardware Diagnostics page. Our page on Hard Drive Connectivity may also be useful by analogy.
Some Sun patches resolve security holes in vold by commenting devices out of the /etc/rmmount.conf file. While this is effective, it also disables the devices that are commented out.
For vold to work with your CD ROM and floppy drives within CDE in the designed fashion, you should uncomment the following lines in your /etc/rmmount.conf:

ident hsfs cdrom
ident ufs floppy cdrom pcmem
ident pcfs floppy pcmem
action cdrom
action floppy
Note that this will allow mounted CDs and floppies to be mounted with SUID permissions. For systems where this is inappropriate, your /etc/rmmount.conf file should also contain the following:

mount * ufs -o nosuid
mount * hsfs -o nosuid
(Note that pcfs does not understand SUID, so pcfs does not pose a risk in the same way that hsfs and ufs do.)
The Solaris Volume Manager documentation contains substantial additional information about the operation of vold.

Solaris 2.x Core Dump Analysis
If you are having trouble getting a core dump, see the savecore page.
Several useful pieces of information can be found by running strings on the vmcore.# file and piping it through more or grep: strings vmcore.# | more. In particular, this will tell you the architecture of the system and the OS level. The message buffer is also near the top of the strings output, and may include messages that had not been written to the console or log files yet. (Note that the message buffer is a ring buffer, so the messages may not be in cronological order.)
If the system panic'ed due to a bad trap, adb can determine the instruction that was running at the time of the crash.
crash can also provide useful information, including lock and kernel memory allocation. crash output is notably less cryptic than adb output.
Some netstat, nfsstat and arp commands are also available for crash dumps. After running the command on the corefiles (eg netstat -d unix.# vmcore.#), compare the output to that on the live system, since some of the options do not report the crash dump statistics.
ipcs can also be used with crash dumps in the following format: ipcs -a -C vmcore.# -N unix.# See the IPC page for information on these facilities.

Changing a hostname
The following steps are required to change a Sun system's hostname.
• /etc/hosts.allow (to correct access permissions)
• /etc/dfs/dfstab on this system's NFS servers (to allow proper mount access)
• /etc/vfstab on this system's NFS clients (so they will point at the correct server)
• kerberos configurations
• ethers and hosts NIS maps
• DNS information
• Netgroup information
• cron jobs should be reviewed.
• Other hostname-specific scripts and configuration files.
Additional steps may be required in order to correct issues involving other systems.
Having said all that, the minumum number of changes required are:
• /etc/nodename
• /etc/hosts
• /etc/hostname.*
• /etc/net/*/hosts
Error Message interpretation
See below for a list of common error messages.
Traps and interrupts can be blocked by a kernel thread's signal mask, or they can trigger an exception handling routine. In the absence of such a routine or mask, the process is terminated.
Traps are syncronous messages generated by the process or its underlying kernel thread. Examples include SIGSEGV, SIGPIPE and SIGSYS. They are delivered to the process that caused the signal.
Trap messages can be discovered in a number of places, including error logs, adb output, and console messages. Sun provides a couple of files that can help determine the type of trap encountered:
• /usr/include/sys/trap.h (software traps)
• /usr/include/v7/sys/machtrap.h (hardware traps, 32 bit)
• /usr/include/v9/sys/machtrap.h (hardware traps, 64 bit)
ECC (Error Checking and Correcting) interrupts are reported as traps when a bit error is corrected. These, while they do not crash the system, are usually a signal that the memory chip in question needs to be replaced.
Critical errors include things like fan/temperature warnings or power loss that require immediate attention and shutdown.
Fatal errors are hardware errors where proper system function cannot be guaranteed. These result in a watchdog reset.
Bus Errors
A bus error is issued to the processor when it references a location that cannot be accessed.
• Illegal address: (usually a software failure)
• Instruction fetch/Data load: (device driver bug)
• DVMA: (on an Sbus system)
• Synchronous/asynchronous data store
• MMU: (Memory Management Unit: can be hardware or software, but frequently are system board problems.)
These notify the CPU of external device conditions that are asynchronous with normal operation. They can be delivered to the responsible process or kernel thread.
In Solaris, interrupts are handled by dedicated interrupt-handling kernel threads, which use mutex locks and semaphores. The kernel will block interrupts in a few exceptional circumstances, such as during the process of acquiring a mutex lock protecting a sleep queue.
• Device done or ready.
• Error detected.
• Power on/off.
Watchdog Reset
Watchdog resets can be caused by hardware or software issues. See the watchdog reset page for information on how to troubleshoot watchdog resets.
Error Message List
A complete (or even reasonably complete) listing of error messages on Solaris is beyond the scope of this site. For that matter, the nature of an evolving operating system may put it beyond the scope of any reasonably sized page. Maybe a wiki? If someone has such a resource, let me know and I will link to it.
Having said that, this page contains a list of several of the most common error messages. Where I have been able to identify a usual cause for an error message, I have included that.
There are several sources that contain listings of error messages that are useful for debugging purposes.
One of the best resources is the Solaris Common Messages and Troubleshooting Guide released by Sun with Solaris 8. Since this is a better resource than I could provide for Solaris up through 8, I have focused on Solaris 10. (There is obviously a lot of overlap.)
The SunSolve web site is available to anyone with a Sun service contract. Its search feature can be used to look up key words in an error message to look for current bug reports and patches that may resolve them. This page does not provide a listing of bug reports or patches to apply for given error messages in certain conditions. This page is intended as a supplement to Sunsolve, not a replacement.
The Intro(2) man page contains an introduction to system calls and error numbers. The information comes from the errno.h include file. Several include files contain at least basic information about different kinds of error messages:
• /usr/include/sys/errno.h (error messages, including abbreviations and numbers seen in truss output.)
• /usr/include/sys/trap.h (software traps)
• /usr/include/v7/sys/machtrap.h (hardware traps, 32 bit)
• /usr/include/v9/sys/machtrap.h (hardware traps, 64 bit)
These messages are alphabetized by the first non-variable portion of the message. Wording may vary slightly between Solaris versions or even patch levels. If you run across common messages not on this list, feel free to make a comment to the Solaris Troubleshooting blog.
• Accessing a corrupted shared library (ELIBBAD): exec(2) was unable to load a required static shared library. The most common cause for this is a corrupted library.
• Address already in use (EADDRINUSE): The protocol does not permit using an address that is already in use. This error indicates a software programming bug.
• Address family not supported by protocol family (EAFNOSUPPORT): The protocol does not support the requested address. This indicates a software programming bug.
• Arg list too long (E2BIG): The argument list includes both the argument list and the environment variable settings. The most common cause for this problem is that so many environment variables are set that it exceeds the size of the argument buffer used by exec(2). The easiest solution may be to unset some environment variables in the calling shell.
• Argument out of domain (EDOM): This error appears when an improper argument is submitted to a math package programming function. (For example, an attempt to take a square root of a negative number would probably yield this error.) It may be helpful to use matherr(3M) to diagnose the problem, or the programmer may need to implement argument-checking before the function is called.
• Arguments too long: This is a C shell message indicating that more than 1706 arguments follow a command. This may happen if globbing is applied to a large number of objects (eg rm * in a directory of more than 1706 objects). Temporarily switching to Bourne shell may resolve the problem, since Bourne shells dynamically allocate space for arguments.
• Assertion failed: This is a result of an assert(3C) debugging command that the programmer inserted into the program. The output will include an expression, a source file number and a code line number. The information may be useful in examining the source code.
• Attachment point not found: Use cfgadm to list available attachment points. Check the physical connection to the desired device.
• Attempting to link in more shared libraries than system limit (ELIBMAX): The executable requires more static libraries than the current system limit.
• authentication receive failed: Initiator unable to receive authentication information. Verify network connectivity to storage device and authentication server.
• authentication transmit failed: Initiator unable to transmit authentication information. Verify network connectivity to storage device and authentication server.
• Bad address (EFAULT): A function taking pointer argument has been passed an invalid address. This may result from supplying the wrong device or option to a command, or it may be the result of a programming bug.
• Bad file number (EBADF): The file descriptor references a file that is either not open or is open for a conflicting purpose. (eg, a read(2) is specified against a file that is open for write(2) or vice-versa.) This is a programming bug.
• Bad module/chip: This error message usually indicates a memory module or chip that is associated with parity errors. This is a hardware fault.
• BAD SUPER BLOCK: Check the Trap 3E entry below to see if there are possible hardware or SCSI configuration causes for this problem. It may be possible to boot from alternate super blocks. If there is no current backup, boot from a CD and back up the raw partition with ufsdump or another similar utility. Solaris 10's 6/06 release includes enhancements to fsck to automatically find and repair bad superblocks. This option should only be used to repair filesystems that were created with mkfs or newfs. For older systems, an alternate superblock can frequently be found with a
newfs -N /dev/rdsk/c#t#d#s#
command while booted from a CD. (Note the -N option. Running this command without this option may mess things up beyond repair.) fsck can be run against an alternate superblock with
fsck -o b=superblock /dev/rdsk/c#t#d#s#
If there is a lot of output, it may be necessary to choose the -y option to avoid having to answer a ton of prompts. We may need to try several alternate superblocks before finding a working one. Once we are done, we need to re-install the bootblock:
cd /usr/platform/`arch -k`/lib/fs/ufs
/usr/sbin/installboot ./bootblk /dev/rdsk/c#t#d#s#
• BAD TRAP: The causes for bad traps include system text errors, data access faults, data alignment errors or some types of user software traps. These can indicate either a hardware fault or a mismatch between the hardware and its software configuration. They may also indicate a CPU with an obsolete firmware. Bad traps usually result in a panic, sync, dump, reboot cycle. The kernel traceback message on the console will frequently indicate the hardware component that generated the bad trap. If the configuration for this component is correct, it will need to be replaced (or at least reseated).
• /bin/sh: ... too big: This Bourne shell message is a variant of Not enough space. Check that message for steps to take.
• Block device required (ENOTBLK): A raw device was specified where a block device is required.
• Broken pipe (EPIPE): No reading process was available to accept a write on the other end of a pipe. This can happen when the reading process (the process after the pipe) exits suddenly.
• Bus Error: I/O was attempted to a device that is unavailable or does not exist. See Bus Error above.
• Cannot access a needed shared library (ELIBACC): Either the library does not exist, the LD_LIBRARY_PATH variable does not include the library, or the user is not permissioned to use it. The library in question can usually be pinned down with truss.
• Cannot assign requested address (EADDRNOTAVAIL): The requested address is not on the current machine.
• Cannot exec a shared library directly (ELIBEXEC): You can't execute shared libraries directly. This error indicates a software bug.
• Cannot install bootblock: On an x86 system, this error typically appears when a newfs and restore operation was carried out without performing a installboot before installing the OS. It may be possible to install the bootblock from the CD drive in single-user mode (note that Sun does not guarantee this procedure):
cd /usr/platform/`arch -k`/lib/fs/ufs

installboot ./pboot ./bootblk /dev/rdsk/c#t#d#s#
• Cannot send after transport endpoint shutdown (ESHUTDOWN): The transport endpoint has been shut down, so data was unable to be sent. The solution is usually to restore the endpoint and re-run the transfer. (We may need to troubleshoot why the remote endpoint became unavailable.)
• can't accept: Initiator does not accept the specified data of the given format. Consult storage device documentation to look for compatibility information for the server hardware and OS.
• can't accept ... in security stage: Device responded with unsupported login information during login security phase. Verify storage device authentication settings. Consult storage device documentation to look for compatibility information for the server hardware and OS.
• can't find environment variable: The specified environment variable has not been set. Check for a typo and/or verify that the variable has been set.
• Can't invoke /etc/init: The init binary is missing or corrupted during a reboot. We may be able to complete the boot by copying init from a CDROM during a CDROM reboot.
• capacity of this LUN is too large: SCSI partitions must be less than 2TB.
• Channel number out of range (ECHRNG): A stream head attempted to open a minor device that is in use or does not exist. We need to make sure that the stream device exists, along with an appropriate number of minor devices, and that it matches the hardware configuration. It may be necessary to schedule jobs differently to allow for limited system resources.
• check boot archive content: If SMF does not start up on its own, this message in response to svcs -x may indicate a failure of svc:/system/boot-archive:default To resolve this problem, select the Solaris failsafe archive option in the GRUB boot menu during the next reboot. The failsafe boot option provides instructions for rebuilding the boot archive. Once that is complete, the boot can be continued by clearing the SMF boot archive with the svcadm clear boot-archive command.
• Command not found: This is a C shell error message that means exactly what it says. It typically means that the command was misspelled or does not live on the PATH.
• Communication error on send (ECOMM): The link between machines breaks after data is sent, but before the confirmation is received.
• Component system is busy, try again: failed to offline: cfgadm attempted to remove or replace a device with a mounted file system, swap area or configured dump device. Unmount the file system, remove the swap and/or disable the dump device, then retry the cfgadm command. See the cfgadm(1M) man page.
• Configuration operation invalid: invalid transition: The incorrect device may have been specified, or there may be a problem with the device or its seating. Use cfgadm to check the receptacle and its state. The card may need to be reseated.
• Connection refused (ECONNREFUSED): The target machine actively refused the connection. The service may not be active, or there may be restrictions on connections (such as the hosts.allow and hosts.deny in TCP wrappers).
• Connection reset (ECONNRESET): The target system forcibly closed an existing connection. This typically happens as a result of a reboot or a timeout.
• Connection timed out (ETIMEOUT): The target host is unreachable due to network problems or the system being down.
• Core dumped: A core file (image of software memory at the time of failure) has been taken. See Core File Management.
• Corrupt label: This happens if cylinder 0 has been overwritten, usually by a database using a raw partition including cylinder 0. The best solution is to back everything up and repartition the disk with cylinder 0 either not in any partition or at least in a partition with a filesystem (such as UFS) that respects cylinder 0.
• cpio: Bad magic number/header: The cpio archive has become corrupted. We can try to recover whatever we can by using the cpio -k command.
• Cross-device link (EXDEV): Hard links are not permitted across different filesystems. Use a soft link instead.
• Data access exception: Mismatch between the operating system and disk storage hardware. This can be due to mis-seated DIMMs or disk problems, so it makes sense to try to identify any hardware problems. Usually, the operating system (and perhaps filesystem) will need to be upgraded to deal with the newer hardware.
• DataDigest=... is required, can't accept: Device returned an improperly processed DataDigest. Verify that storage device digest settings are compatible with the initiator.
• Data Fault: This is a particular type of bad trap that indicates a configuration text or data access fault. See BAD TRAP above.
• Deadlock situation detected/avoided (EDEADLK): A potential deadlock over a system resource (usually a lock) was detected and avoided. The software should be examined to see if it can be made more resilient.
• Destination address required (EDESTADDRREQ): An address was omitted from an operation that requires one.
• /dev/fd/#: cannot open: Indicates that the file descriptor file system (fdfs) is not mounted correctly. In most cases, the problem is that it is mounted either nosuid or not at all. The file descriptor file system should have the following options in the vfstab:
fd - /dev/fd fd - no -
• Device busy (EBUSY): A hard drive or removable media failed to unmount or eject due to an active process using them. The fuser command allows us to see what processes are using the filesystem or even kill them with a command like:
fuser -ck /mountpoint
(Make sure that you know what processes are running on a filesystem before killing them.)
• DIMMs Manufacturer Mismatch: DIMMs in the system are not on the hardware compatibility list.
• Directory not empty: This is an error from rmdir which means exactly what it says. Non-empty directories cannot be removed. (If a process is holding a file open, it is possible to track down the culprit by looking for the inode of the file in question (ls -i filename) in pfiles output.)
• Disc quota exceeded (EDQUOT): A user's disk quota has been exceeded. Some of the user's files can be removed or the quotas can be increased with edquota.
• Disk# not unique: This error is displayed if there are multiple EEPROM devalias entries for a disk. At the ok> prompt, the values of the aliases can be shown with
ok> printenv
the aliases can be reset with
ok> nvunalias disk#
ok> nvalias disk# device-path
• dquot table full: The UFS quota table needs to be increased in size. This is done by increasing ndquot in /etc/system and rebooting. ndquot defaults to (maxusers x 40)/4 + max_nprocs
• dr in progress: This error may occur if a SCSI unconfigure operation fails while only partially completed. The controller may need to be reconfigured with cfgadm
• driver not attached: No driver currently attached to the specified device because no device exists at the node or the device is not in use. This may or may not mean that a proper driver is not installed. Make sure that the driver is installed and properly configured.
• empty RADIUS shared secret: The RADIUS shared secret needs to be set.
• Error 88 (EILSEQ): This is an illegal byte sequence error. Multiple characters have been provided where only one is expected.
• Error code 2: access violation: This error is due to a permissioning or pathing error on a tftp get.
• Error: missing file arg (cm3): A filename was not included in an sccs command that requires one.
• error opening dir: The specified path may not be a directory.
• error writing name when booting: /etc/nodename must contain exactly one line with the name of the system and no blanks or returns.
• esp0: data transfer overrun: This error appears when we attempt to mount a CD drive with an 8192 block size as opposed to the Sun-standard 512 block size. Check with the drive manufacturer to see if the block size can be switched.
• ether_hostton: Bad file number/Resource temporarily unavailable: These messages may be a result of a mis-matched nodename file. Make sure that the /etc/nodename entry matches the corresponding /etc/hostname.interface and /etc/inet/hosts files.
• Event not found: The shell reports that a command matching the request cannot be found in the history buffer for the shell session. The history command shows the current contents of the history buffer.
• Exec format error (ENOEXEC): This error usually means that the software was compiled for an architecture other than the one on which it finds itself. This may also happen if an expected binary compatibility package is not installed. The file command displays the expected architecture for the binary.
• Failed to initialize adapter: If the adapter has been correctly identified, this means that the configuration of the adapter is incorrect. In particular, make sure to check the DMA settings.
• Failed to receive login response: Initiator failed to receive a login Payload Data Unit (PDU) across the network. Verify that the network connection is working.
• Failed to transfer login: Initiator failed to transfer a login Payload Data Unit (PDU) across the network. Verify that the network connection is working.
• Fast access mmu miss: This is usually due to a hardware problem. Memory is a possible culprit, as are the system board and CPU. Check PROM Monitor Diagnostics for hardware diagnostics on OBP/Sparc systems.
• File descriptor in bad state (EBADFD): The requested file descriptor does not refer to an open file or it refers to a file descriptor that is restricted to another purpose. (For example, a read request is made to a file descriptor that is open for writing only.)
• File exists (EEXISTS): An existing file was targeted for a command that would have overwritten it improperly. For example, there may have been a request to overwrite a file while the csh noclobber option is set, or there may have been a request to set a link to the name of an existing file.
• File locking deadlock (EDEADLOCK): Two processes deadlocked over a resource, such as a lock. This is a software programming bug.
• File name too long (ENAMETOOLONG): The referenced file name is longer than the limit specified in /usr/include/limits.h.
• File system full: The file system is full. (Error messages sometimes mean what they say.) If the message occurs during a login, the problem is likely the filesystem that includes the utmpx file (usually /var).
• File too large (EFBIG): The file size has grown past what is allowed by the protocol or filesystem in question, or exceeds the resource limit (rlimit) for file size. The resource limit can be checked by running ulimit -a in Bourne or Korn shells or limit in C shell. Check the Resource Management page for additional information on managing resource limits.
• Giving up: In the context of a SCSI command, this means that the timeout has been exceeded. This is usually due to a hardware or connection problem, but it can be caused by contention on the SCSI channel, or even a mis-match in timeout settings between the OS and the device in question.
• Hardware address trying to be our address: Either we have two systems on our network with the same IP address, or we have snooping enabled on a device on the network.
• Host is down (EHOSTDOWN): A connection attempt failed because the target system was unavailable.
• HeaderDigest=... is required, can't accept: Device returned an improperly processed HeaderDigest. Verify that storage device digest settings are compatible with the initiator.
• Host name local configuration error: sendmail wants to have a fully qualified domain name for the local host. It is good practice to include a fully qualified domain name in the hosts file entry for the local server.
• Hypertransport Sync Flood occurred on last boot: Uncorrectable ECC error caused the last reboot. For x64 systems, check the service processor's System Event Log and BIOS log to identify the culprit.
• Identifier removed (EIDRM): There is a problem accessing a file associated with messaging, semaphores or shared memory. Check the msgctl(2), semctl(2) or shmctl(2) man page for more details.
• ieN Ethernet jammed: The number of successive failed transmission attempts has exceeded the threshold. Check whether the network is saturated or check for other network problems.

• ieN no carrier: The carrier detect pin died during a packet transmission, resulting in a dropped packet. Check for loose connections and otherwise check the network.
• If pipe/FIFO, don't sleep in stream head (ESTRPIPE): There is a problem with the STEAMS connection.
• ifconfig: bad address: Check /etc/hostname.* to make sure that the entries match the hosts file. When this error occurs early in the boot process, make sure that the filesystem containing hostname.* and hosts is online at that stage of the boot process. If “files” is not the first entry in the “hosts” line of /etc/nsswitch.conf, the hostname lookup will not be possible until the interface comes online.
• ifconfig: no such interface: Make sure that the /etc/hostname.interface file exists.
• Illegal instruction: This error message means exactly what it says. This may come about because the binary is not compiled for this architecture (see “Exec format error” above), or it may come as a result of trying to run a data file as a program. If this appears during a boot, it means that the system is trying to boot from a non-boot device, that the boot information has become corrupted, or that the boot information is meant for a different architecture.
• Illegal seek (ESPIPE): There is a problem with a pipe in the statement. A workaround suggested by Sun is to redirect the output of the source command to a scratch file, then process the file.
• Initiator could not be successfully authenticated: Verify CHAP and/or RADIUS settings, as appropriate.
• Initiator is not allowed access to the given target: Verify initiator name, masking and provisioning.
• initiator name is required: The initiator name is improperly configured.
• Interrupted system call (EINTR): An signal (like an interrupt or quit) was received before the system call had completed. (If we try to resume, we may error out as a result of this condition.)
• Invalid argument (EINVAL): System cannot interpret a supplied parameter. Depending on the context, this may be an indication that the object named by the parameter is not set up properly.
• Invalid null command: This may indicate that there were two pipes in a row (“||”) in the referenced command.
• I/O error (EIO): This references a physical I/O fault. Depending on the context, it makes sense to replace the removable media, check all connections, run diagnostics on the referenced hardware or fsck the filesystem. If this error occurs during a write, we must assume that the data is corrupt.
• Is a directory (EISDIR): We tried to treat a directory like a file.
• iSCSI service or target is not currently operational: Run diagnostics on the storage device hardware; check storage device software configuration.
• Kernel read error: savecore is unable to read the kernel data structures to produce a crash dump. This may indicate a hardware problem, especially a memory problem. This problem may accompany a BAD TRAP error.
• Killed: This may happen as a result of a memory allocation attempt where either there is insufficient swap space or the stack and data segment size are in conflict. A “Killed” message may also appear when a program is sent a SIGKILL by other means, such as a kill command.
• kmem_free block already free: This is a software programming bug, probably in a device driver.
• fatal: can't set protection on segment: Sun reports a case where this error occurred due to a lack of swap space. complained because there was no segment on which to set protections.
• fatal: open failed: No such file or directory: The linker was unable to find the shared library in question. Make sure that LD_LIBRARY_PATH is set properly.
• fatal: relocation error: referenced symbol not found: The symbol referenced by the specified application was not found. This error most frequently occurs after installations or upgrades of shared libraries. ldd -d on the application will show its dependencies. Depending on the nature of the conflict, it may be resolvable by changing the LD_LIBRARY_PATH or installing an appropriate version of the shared library.
• Link has been severed (ENOLINK): The connection to a remote machine has been severed, either by the remote process dying or a network problem.
• Login incorrect: This error means that an appropriate username and password pair was not entered. This may be due to a problem with the passwd and shadow file, the naming service, or the user forgetting login credentials.
• login redirection failed: Storage device attempted to redirect initiator to an invalid destination. Verify storage device redirection settings.
• Memory Configuration Mismatch: Can be caused by damaged or unsupported DIMMs, or by running non-identical DIMMs within the same bank.
• Message too long (EMSGSIZE): A message was sent that was larger than the internal message buffer.
• Miscellaneous iSCSI initiator errors: Check the initiator.
• Missing parameters (e.g, iSCSI initiator and/or target name): Verify that the initiator and target name are properly specified.
• mount: ...already mounted... (EBUSY): Either the filesystem is mounted elsewhere, an active process has its working directory inside the mount point or the maximum number of mounts has been exceeded.
• mount: giving up on...: The remote mount request was unsuccessful for more than the threshold number of retries. Check the network connection and make sure that the NFS server is sharing the directory to the client as expected.
• mount: mount-point...does not exist: The directory specified as the mount point does not exist.
• mount: the state of /dev/dsk/... is not okay: The filesystem should either be mounted read-only or fsck-ed.
• Network dropped connection because of reset (ENETRESET): The remote host crashed or rebooted.
• Network is down (ENETDOWN): A transport connection failed due to a dead network.
• Network is unreachable (ENETUNREACH): Either there is no route to the network, or negative status information was received from intermediate network devices.
• NFS getattr failed for server...RPC: Timed out: The NFS server has failing hardware. (For a server that is slow to respond, the NFS server not responding message would appear instead.)
• nfs mount: Couldn't bind to reserved port: The NFS server has multiple network cards bound to the same subnet.
• nfs mount: mount:...Device busy: An active process has a working directory inside the mount point.
• NFS mount:...mounted OK: A backgrounded mount completed successfully. This may be an indication that the server response is poor, since otherwise the mount would have completed immediately and not required backgrounding.
• NFS read failed for server: This is a permissions problem error message. In addition to checking the permissions on the NFS server, make sure that the permissions underneath the mount are acceptable. (Mount points should have 755 permissions to avoid odd permissioning behavior on mounted filesystems.)
• nfs_server: bad getargs: The arguments are unrecognized or incorrect. This may be an indication of a network problem, or it may indicate a software configuration problem on the client.
• NFS server ... not responding: The network connection to the NFS server is either slow or broken.
• NFS server ... ok: The network connection to the NFS server has been restored. This is a followup to NFS server ... not responding.
• nfs umount: ... is busy: An active process has a working directory inside the specified NFS mount. See the Device busy error message.
• NFS write error on host ... No space left on device: If an NFS mount runs out of space, attempts to write to files on the share may corrupt or zero out those files.
• NFS write failed for server ... RPC: Timed out: The filesystem is soft mounted, and response time is inadequate. Sun recommends that writable filesystems not be soft-mounted, as it can lead to data corruption.
• No carrier-cable disconnected or hub disabled?: This error may manifest due to a physical networking problem or a configuration issue.
• No child processes (ECHILD): An application attempted to communicate with a cooperating process that does not exist. Either the child exited improperly or failed to start.
• No default media available: Drives contain no floppy or CD media to eject.
• No directory! Logging in with home=/: The home directory either does not exist or is not permissioned such that the user can use it. If home directories are automounted, it may be necessary to troubleshoot the automounter.
• no driver found for device: A driver has been disabled while the device is still attached. Depending on the type of device, cfgadm, drvconfig, devfsadm or a reconfiguration reboot (boot -r) may be required. Check the System Administration Guide: Devices and File Systems document.
• No message of desired type (ENOMSG): Something attempted to receive a message of a type that does not exist on the message queue. See the msgsnd(2) and msgrcv(2) man pages.
• No record locks available (ENOLCK): Any of several different locking subsystems, including fcntl(2), NFS lockd and mail, may yield this message when no more locks are available.
• No route to host (EHOSTUNREACH): In practice, this message is not distinguishable from Network is unreachable.
• No shell Connection closed: The shell specified for the user is either unavailable or illegal. Make sure it is listed in /etc/shells and that it exists. It may be necessary to change the passwd entry for this user to assign a valid shell.
• No space left on device (ENOSPC): The disk, tape or diskette is full.
• No such device (ENODEV): An operation was attempted on an inappropriate or nonexistent device. Make sure that it exists in /devices and /dev. The drvconfig or boot -r commands can be used to regenerate many /devices entries.
• No such device or address (ENXIO): I/O has been attempted to a device that does not exist or that exists beyond the limits of the device. Make sure that the device in question is powered up and connected properly, including the correct SCSI ID.
• No such file or directory (ENOENT): The file or path name does not exist on the system. Make sure that the relevant filesystems are mounted and that the expected files and/or directories exist.
• No such process (ESRCH): The process does not exist on the system. It may have finished prior to the attempt to reference it.
• No such user ... cron entries not created: Even though a file exists in /var/spool/cron/crontabs for this username, the username is not present in the passwd database.
• No utmpx entry: The filesystem containing the utmpx file is full. This may need to be resolved in single-user mode, since logins will not be permitted.
• Not a data message (EBADMSG): Data has come to the head of a STREAMS queue that cannot be processed. See the man pages for read(2), getmsg(2) and ioctl(2).
• Not a directory (ENOTDIR): A non-directory was specified as an argument where a directory is required.
• Not a stream device (ENOTSTR): The file descriptor used as a target for the putmsg(2) or getmsg(2) is not a STREAMS device.
• Not a UFS filesystem: The boot device is improperly defined. For x86, boot the system with the Configuration Assistant/boot CD and identify the disk from which to boot. For PROM-based systems, set the boot-device properly in the PROM environment variables.
• Not enough space (ENOMEM): Insufficient swap space available.
• Not found: The specified command could not be found. Check the spelling and the PATH.
• Not login shell: Use exit to get out of non-login shells. (The logout command can only be used from login shells.)
• Not on system console: Direct root logins are only permitted on the system console unless otherwise specified in /etc/default/login.
• Not owner (EPERM): Action attempted that can only be performed by object owner or the superuser.
• Not supported (ENOTSUP): A requested application feature is not available in the current version, though it may be expected in a future release.
• Object is remote (EREMOTE): We tried to share a resource not on the local machine.
• Operation already in progress (EALREADY): An operation was already in progress on a non-blocking object.
• Operation canceled (ECANCELED): The asynchronous operation was canceled before completion.
• Operation not applicable (ENOSYS): No system support exists for this operation.
• Operation not supported on transport endpoint (EOPNOTSUPP): Tried to accept a connection on a datagram transport endpoint.
• Operation now in progress (EINPROGRESS): Operation in progress on a non-blocking object.
• Option not supported by protocol (ENOPROTOOPT): A bad option or level was specified.
• Out of memory: System is running out of virtual memory (including swap space). See “Not enough space” as well.
• Out of stream resources (ENOSR): No STEAMS queues or no STREAMS head data structures available during a STREAMS open.
• Overlapping swap volume: Make sure that the additional swap volumes have unique names.
• Package not installed (ENOPKG): The attempted system call belongs to a package that is not installed on this system.
• Paired DIMMs Mismatch: Checksum mismatch between two DIMMs in a pair. Can be caused by damaged or non-identical DIMMs.
• Panic – boot: Could not mount filesystem: (During a Jumpstart) The Jumpstart boot process is unable to get to the install image. Make sure that the Jumpstart configurations and file shares are correct.
• Panic ... valloc'd past tmpptes: May occur if maxusers is set to an absurdly high number. It should not be set past the number of MB of RAM or 4096, whichever is smaller.
• Permission denied (EACCES): The attempted file access is forbidden due to filesystem permissions.
• Protocol family not supported (EPFNOSUPPORT): The protocol has not been implemented on this system.
• Protocol not supported (EPROTONOSUPPORT): The protocol has not been configured for this system. Check the protocols database (/etc/inet/protocols by default).
• Protocol wrong type for socket (EPROTOTYPE): Application programming error or misconfigured protocols. The requested protocol does not support the requested socket type. Make sure that the protocols database matches with the corresponding entries in /usr/include/sys/socket.h.
• quotactl: open Is a directory: A directory named “quota” can cause edquota to fail. Such directories should be renamed.
• RADIUS packet authentication failed: Re-set the RADIUS shared secret.
• Read error from network: Connection reset by peer: The remote system crashed or rebooted during an rsh or rlogin session.
• Read-only file system (EROFS): We can't change stuff on filesystems that are mounted read-only.
• received invalid login response: Storage device response was unexpected. Verify initiator authentication settings.
• Requested iSCSI version range is not supported by the target: The initiator's iSCSI version is not supported by the target storage device. Check the compatibility lists. See if firmware or driver upgrades would be sufficient.
• Requested ITN does not exist at this address: The iSCSI target name (ITN) is not accessible. Verify the initiator discovery information and storage device configuration.
• Requested ITN has been removed and no forwarding address is provided: The requested iSCSI target name is no longer accessible. Verify the initiator discovery information and storage device configuration.
• Resource temporarily unavailable (EAGAIN): fork(2) cannot create a new process due to a lack of resources. These resources may include limits on active processes (see the Resource Management page) or a lack of swap space.
• Restartable system call (ESTART): The system call has been interrupted in a restartable state.
• Result too large (ERANGE): This is a programming or data input error. The result of a calculation is not representable in the defined data type. The matherr(3M) facility may be helpful in debugging the problem.
• ROOT LOGIN ...: Someone has just logged in as root or su-ed to root.
• RPC: Program not registered: Make sure that the requested service is available.
• rx framing error: This error usually indicates a problem with the network hardware. Framing errors are types of CRC errors, which are usually caused by physical media problems.
• SCSI bus DATA IN phase parity error: This is a problem related to SCSI hardware or connections. It may have to do with hardware that is not qualified for attachment to Sun servers, connections with cables that are flaky or too long (total length more than 6 meters), bad terminators or flaky power supplies. See the SCSI transport failed: reason 'reset' message as well.
• SCSI transport failed: reason 'reset': The system sent data that was never received due to a SCSI reset. This may occur due to conflicting SCSI IDs, hardware that is not qualified for attachment to Sun servers, connections with cables that are flaky or too long (total length more than 6 meters), bad terminators or flaky power supplies. These issues have also been observed on systems where the highest capacity DIMMs are not in the lowest numbered slots. Disk arrays wth read-ahead caches can sometimes also cause this problem; turn off the caching to see if the problem goes away. Non-obvious SCSI ID conflicts may be diagnosed using the PROM monitor probe-scsi-all command. (See OBP Command Line Diagnostics for more details.) These errors may also happen when the SCSI device and the server are set to different SCSI timeout thresholds.
• Segmentation Fault: These can be produced as a result of programming errors or improperly set rlimit resource settings. (See Resource Management for how to check and adjust resource settings.) Segmentation faults are an indication that the program has attempted to access an area of memory that is protected or does not exist. Programming causes for segmentation faults include dereferencing a null pointer and indexing past the bounds of an array.
• setmnt: Cannot open /etc/mnttab for writing: The system is unable to write to /etc/mnttab. This may be caused by the /etc directory being mounted read-only (which can happen during certain types of boot problems).
• share_nfs: /home: Operation not applicable: A local filesystem is mounted on /home, which is usually reserved for use by the automounter.
• skipping LIST command – no active base: A LIST command is present without an associated BASE command. (cachefspack)
• Socket type not supported (ESOCKTNOSUPPORT): The socket type's support has not been configured for this system.
• Soft error rate ... during writing was too high: The number of soft errors on a tape device have exceeded the threshold. It may be due to a dirty head, bad media or a faulty tape drive.
• Software caused connection abort (ECONNABORTED): The connection was aborted within the local host machine.
• Stale NFS file handle (ESTALE): The file or directory on the NFS server is no longer available. It may have been removed or replaced. A remount may be needed to force a renegotiation of file handles.
• statd: cannot talk to statd: statd has left remnants in the /var/statmon/sm and /var/statmon/sm.bak directories. Files named after inactive hosts should be removed, and statd and lockd should be restarted.
• su: No shell: The default shell for root is improper. It may have been set to a nonexistent program or an illegal shell. This problem has been known to occur when an extra space is appended to the “root” line of the passwd file. The passwd file will need to be repaired while booted from CDROM or network.
• syncing file system: The kernel is updating the superblocks before taking the system down or in the wake of a panic.
• System booting after fatal error FATAL: This can be caused by UPA address parity errors, Master queue overflows or DTAG parity errors. This is going to be due to a bad CPU or possibly a bad system board.
• tar: ...: No such file or directory: The specified target (which defaults to TAPE) is not available. This may be due to a hardware problem with the tape drive or connections, or to a misspecified target.
• tar: directory checksum error: The checksum of the files read from tape do not match the checksum in the header block. This may be due to an incorrectly specified block size or a bad piece of tape media.
• tar: tape write error: A physical write error has occurred on the tar target.
• Target hardware or software error: Run diagnostics on the storage device hardware; check storage device software configuration.
• Target has insufficient session, connection or other resources: Check storage device settings. Check with storage device vendor to see if resource settings can be increased or capacity can be otherwise increased.
• target protocol group tag mismatch: Initiator and target had a Target Portal Group Tag (TPGT) mismatch. Verify TPGT discovery settings on initiator and storage device.
• Text file busy (ETXTBSY): An attempt was made to execute a file that was open for writing.
• The SCSI bus is hung: The likely cause is a conflict in SCSI target numbers. See the SCSI transport failed: reason 'reset' message as well.
• Timeout waiting for ARP/RARP packet: Indicates a network connection problem while booting from the network. This problem can sometimes be observed on subnets containing multiple servers willing to answer a RARP request, which can result in a server without a bootparams file receiving a request. (We have had good luck moving Jumpstart targets to an isolated subnet for initial installations.)
• Timer expired: The timer for a STREAMS ioctl has expired. The cause is device specific, and may be related to a flaky hardware, driver failure or an inappropriately short timeout threshold.
• Too many links (EMLINK): A file has too many hard links associated with it. Use soft links instead.
• Too many open files (EMFILE): A process has exceeded the limit on the number of open files per process. (See the Resource Management page for methods to monitor and manage these limits.)
• Transport endpoint is already connected (EISCONN): Connection request made on an already connected transport endpoint.
• Transport endpoint is not connected (ENOTCONN): The endpoint is not connected and/or an address was not specified.
• Trap 3E: These are caused by a bad boot disk superblock. This may have been caused by a failing disk, faulty disk connections, software misconfiguration or duplicate SCSI addresses. Check the possible hardware and SCSI configuration issues before attempting to recover the superblock using the methods listed under BAD SUPER BLOCK above.
• Too Many Arguments: This is a variant of the C shell's Arguments too long message, except that this time the problem may be the number rather than the length of arguments.
• unable to connect to target: Initiator unable to establish a network connection. This message typically accompanied by an error number from /usr/include/sys/errno.h.
• unable to get shared objects: The executable may be corrupt or in an unrecognized format.
• unable to initialize authentication: Verify that initiator authentication settings are properly configured.
• unable to make login pdu: Initiator could not make a login Payload Data Unit (PDU) based on the initiator and storage device settings. Reset target login parameters and other settings as required.
• unable to schedule enumeration: Initiator unable to enumerate the LUNs on the target. LUN enumeration can be forced via the devfsadm -i iscsi command.
• unable to set [authentication|ipsec|password|remote authentication|username]: Verify that initiator authentication settings are properly configured.
• uname: error writing name when booting: /etc/nodename must contain exactly one line with the name of the system and no blanks or returns.
• Unknown service: Either the service is not listed in the services database (/etc/services by default), or the permissions for the services database are set so that the user cannot read it.
• Value too large for defined data type (EOVERFLOW): Argument improperly formatted for the structure allocated to it.
• WARNING: /tmp: File system full, swap space limit exceeded: Virtual memory has filled up. A reboot is recommended after we have figured out which process is hogging all the memory and/or swap, since the system may be in an unstable state.
• WARNING: TOD clock not initialized: It is likely that the system clock's battery is dead.
• Watchdog Reset: This usually indicates a hardware problem. (See the Watchdog Resets page for a complete discussion.)
• Window Underflow: These errors sometimes accompany a trap, especially at boot time. Some program attempted access of a register window that was not accessible from that processor. These errors may occur when differently sized DIMMs are improperly used together, or when cache memory has gone bad. If mismatched memory is not the problem, the CPU or system board will need to be replaced.
• wrong magic number: See “Corrupt label” above.
• you are not authorized to use: A configuration file (eg at.deny or cron.deny) forbids access to this service.
File Descriptors
A file descriptor is a handle created by a process when a file is opened. A new descriptor is created each time the file is opened. It is associated with a file object which includes information such as the mode in which the file was opened and the offset pointer where the next operation will begin. This information is called the context of the file.
File descriptors are retired when the file is closed or the process terminates. Opens always choose the lowest-numbered file descriptor available. Available file descriptors are allocated as follows:
• rlim_fd_cur: It is dangerous to set this value higher than 256 due to limitations with the stdio library. If programs require more file descriptors, they should use setrlimit directly.

• rlim_fd_max: It is dangerous to set this value higher than 1024 due to limitations with select. If programs require more file descriptors, they should use setrlimit directly.
Kernel memory size can be tracked using the sar -k command. The total of the "alloc" fields is the kernel memory size. If it appears to be growing without bound, you may have a memory leak. It should be noted that not all buckets are tracked by sar -k, so the reported memory size is not as accurate as that reported by crash.
On occasion there are problems related to memory leaks in the kernel or one of the associated modules. In these cases, kmastat can provide useful information pinpointing the source of the leak.
To check on kernel memory allocations on a running system, run a crash session as follows:

# crash
dumpfile = /dev/mem, namelist = /dev/ksyms, outfile = stdout
> kmastat
The first number on the "Total" line represents the total amount of memory allocated by the kernel. If this is a significant fraction of available system memory and growing, there is a problem.
The output from the kmastat command also contains information on a number of "buckets" or categories for memory allocation.
Additional information can be obtained via the kmausers command, but this requires that we load kadb prior to booting. To do this, reach the ok> prompt, then:

ok> boot kadb -d
kadb: (hit the "return" key)
kadb[0]: kmem_flags/W 01
kadb[0]: :c
Loading kadb this way means that kadb will only be effective for this current boot session.
Once the system is up, we can either force a core dump via STOP-A/ ok> sync, or we can examine the live system. In either case, inside the crash session we would type:

>kmausers bucket_name
The result will show memory allocations inside that bucket. The names of functions inside each allocation will be a tip-off to what is actually grabbing the memory allocation.
A script can be run from cron to capture this information. The format of this script would be something like:

date >> log_file
echo "kmastat" | /usr/sbin/crash -w log_file
sleep 20
echo "kmausers kmem_alloc_2048" | /usr/sbin/crash -w log_file
Slab Allocator
Solaris 2.4+ uses a kernel memory allocator known as a slab allocator.
A kernel memory allocator performs the following functions:
• Allocate memory
• Initialize objects/structures
• Use objects/structures
• Deconstruct objects/structures
• Free memory
The structures in the memory objects include sub-objects such as linked list headers, mutexes, reference counts and condition variables. In the case of Solaris, the deconstruction step includes setting objects to their initial settings, which can save time when the memory objects have to be re-initialized.
A translation lookaside buffer (TLB) is an associative cache of recent address translations. When the MMU (memory management unit) cannot find a translation in the TLB, it lookus it up in the address maps and loads the address into the TLB. Entries in the TLB are replaced on a least recently used basis.
The slab allocator is organized as a collection of object caches. Each of these caches contains only one type of object (proc structures, vnodes, etc). The kernel is responsible for restoring each object to its initial state when it is released. When a cache requires additional space, the allocator gives a slab of memory from the page-level allocator and creates objects from it. The slab contains enough memory for several object instances. A small part of the slab is used by the cache to manage memory in the slab; the rest is divided into buffers that are the size of the object. The allocator then initializes these buffers with the appropriate constructor.
When the page-level allocator needs to recover memory, unused slabs are reaped by deconstructing the objects on slabs whose objects are all free, then removing the slab from the cache in question.
The structure for each slab includes unused space at the beginning of the slab (coloring area), the set of objects, more unused space (the amount left over after the maximum number of objects has been created), and a slab data area. Each object also includes a four byte area for a free list pointer. The slab data area includes a count of in-use objects, pointers for a doubly-linked list of slabs in the same cache, and a pointer to the first free area in the slab. The coloring areas are different sizes for each slab in a cache (where possible). This allows a balanced distribution of traffic on the hardware caches and memory busses by varying the offsets for the different slabs.
Large object slabs are slightly different in that management data is stored in a separate pool of memory, since large slabs are usually multiples of a page in size. A hash table is also maintained to provide lookups between the management area and the slabs.

Kernel Modules
• modinfo prints out information on all currently loaded kernel modules.
• modload loads kernel modules.
• modunload unloads kernel modules.
• forceload in the /etc/system file loads a module at boot time.
NFS Troubleshooting
Sun's web pages contain substantial information about NFS services; search for an NFS Administration Guide or NFS Server Performance and Tuning Guide for the version of Solaris you are running. The share_nfs man page contains specific information about export options.
If NFS is not working at all, try the following:
• Make sure that the NFS server daemons are running. In particular, check for statd, lockd, nfsd and rarpd. If the daemons are not running, they can be started by running /etc/init.d/nfs.server start. See Daemons below for information on NFS-related daemons.
• Check the /etc/dfs/dfstab and type shareall.
• Use share or showmount -e to see which filesystems are currently exported, and to whom. showmount -a shows who the server believes is actually mounting which filesystems.
• Make sure that your name service is translating the server and client hostnames correctly on both ends. Check the server logs to see if there are messages regarding failed or rejected mount attempts; check to make sure that the hostnames are correct in these messages.
• Make sure that the /etc/net/*/hosts files on both ends report the correct hostnames. Reboot if these have to be edited.
If you are dealing with a performance issue, check
• Network Issues
• CPU Useage
• Memory Levels
• Disk I/O
• Increase the number of nfsd threads in /etc/init.d/nfs.server if the problem is that requests are waiting for a turn. Note that this does increase memory useage by the kernel, so make sure that there is enough RAM in the server to handle the additional load.
• Where possible, mount filesystem with the ro option to prevent additional, unnecessary attribute traffic.
• If attribute caching does not make sense (for example, with a mail spool), mount the filesystem with the noac option. If nfsstat reports a high getattr level, actimeo may need to be increased (if the attributes do not change too often).
• nfsstat reports on most NFS-related statistics. The nfsstat page includes information on tuning suggestions for different types of problems that can be revealed with nfsstat.
If these steps do not resolve the issue, structural changes may be required:
• cachefs can be used to push some of the load from the NFS server onto the NFS clients. To be useful, cfsadmin should be used to increase maxfilesize for the cache to a value high enough to allow for the caching of commonly-used files. (The default value is 3 Mb.)
NFS Client
When a client makes a request to the NFS server, a file handle is returned. The file handle is a 32 byte structure which is interpreted by the NFS server. Commonly, the file handle includes a file system ID, inode number and the generation number of the inode. (The latter can be used to return a "stale file handle" error message if the inode has been freed and re-used between client file accesses.)
If a response is not received for a request, it is resent, but with an incremented xid (transmission ID). This can happen because of congestion on the network or the server, and can be observed with a snoop session between server and client.
The server handles retransmissions differently depending on whether the requests are idempotent (can be executed several times without ill effect) or nonidempotent (cannot be executed several times). Examples of these would include things like reads and getattrs versus writes, creates and removes. The system maintains a cache of nonidempotent requests so that appropriate replies can be returned.
The following daemons play a critical role in NFS service:
• biod: On the client end, handles asynchronous I/O for blocks of NFS files.
• nfsd: Listens and responds to client NFS requests.
• mountd: Handles mount requests.
• lockd: Network lock manager.
• statd: Network status manager.

Solaris Processes
The process is one of the fundamental abstactions of Unix. Every object in Unix is represented as either a file or a process. (With the introduction of the /proc structure, there has been an effort to represent even processes as files.)
Processes are usually created with fork or a less resource intensive alternative such as fork1 or vfork. fork duplicates the entire process context, while fork1 only duplicates the context of the calling thread. This can be useful (for example), when exec will be called shortly.
Solaris, like other Unix systems, provides two modes of operation: user mode, and kernel (or system) mode. Kernel mode is a more privileged mode of operation. Processes can be executed in either mode, but user processes usually operate in user mode.
Per-process Virtual Memory
Each process has its own virtual memory space. References to real memory are provided through a process-specific set of address translation maps. The computer's Memory Management Unit (MMU) contains a set of registers that point to the current process's address translation maps. When the current process changes, the MMU must load the translation maps for the new process. This is called a context switch.
The MMU is only addressable in kernel mode, for obvious security reasons.
The kernel text and data structures are mapped in a portion of each process's virtual memory space. This area is called the kernel space (or system space).
In addition, each process contains these two important kernel-owned areas in virtual memory: u area and kernel stack. The u area contains information about the process such as information about open files, identification information and process registers. The kernel stack is provided on a per-process basis to allow the kernel to be re-entrant. (ie, several processes can be involved in the kernel, and may even be executing the same routine concurrently.) Each process's kernel stack keeps track of its function call sequence when executing in the kernel.
The kernel can access the memory maps for non-current processes by using temporary maps.
The kernel can operate in either process context or system (or interrupt) context. In process context, the kernel has access to the process's memory space (including u area and kernel stack). It can also block the current process while waiting for a resource. In kernel context, the kernel cannot access the address space, u area or kernel stack. Kernel context is used for handling certain system-wide issues such as device interrupt handling or process priority computation.
Additional information is available on the Process Virtual Memory page.
Process Context
Each process's context contains information about the process, including the following:
• Hardware context:
o Program counter: address of the next instruction.
o Stack pointer: address of the last element on the stack.
o Processor status word: information about system state, with bits devoted to things like execution modes, interrupt priority levels, overflow bits, carry bits, etc.
o Memory management registers: Mapping of the address translation tables of the process.
o Floating point unit registers.
• User address space: program text, data, user stack, shared memory regions, etc.
• Control information: u area, proc structure, kernel stack, address translation maps.
• Credentials: user and group IDs (real and effective).
• Environment variables: strings of the form variable= value.
During a context switch, the hardware context registers are stored in the Process Control Block in the u area.
The u area includes the following:
• Process control block.
• Pointer to the proc structure.
• Real/effective UID/GID.
• Information regarding current system call.
• Signal handlers.
• Memory management information (text, data, stack sizes).
• Table of open file descriptors.
• Pointers to the current directory vnode and the controlling terminal vnode.
• CPU useage statistics.
• Resource limitations (disk quotas, etc)
The proc structure includes the following:
• Identification: process ID and session ID
• Kernel address map location.
• Current process state.
• Pointers linking the process to a scheduler queue or sleep queue.
• Pointers linking this process to lists of active, free or zombie processes.
• Pointers keeping this structure in a hash queue based on PID.
• Sleep channel (if the process is blocked).
• Scheduling priority.
• Signal handling information.
• Memory management information.
• Flags.
• Information on the relationship of this process and other processes.
Kernel Services
The Solaris kernel may be seen as a bundle of kernel threads. It uses synchronization primitives to prevent priority inversion. These include mutexes, semaphores, condition variables and read/write locks.
The kernel provides service to processes in the following four ways:
• System Calls: The kernel executes requests submitted by processes via system calls. The system call interface invokes a special trap instruction.
• Hardware Exceptions: The kernel notifies a process that attempts several illegal activities such as dividing by zero or overflowing the user stack.
• Hardware Interrupts: Devices use interrupts to notify the kernel of status changes (such as I/O completions).
• Resource Management: The kernel manages resources via special processes such as the pagedaemon.
In addition, some system services (such as NFS service) are contained within the kernel in order to reduce overhead from context switching.
An application's parallelism is the degree of parallel execution acheived. In the real world, this is limited by the number of processors available in the hardware configuration. Concurrency is the maximum acheivable parallelism in a theoretical machine that has an unlimited number of processors. Threads are frequently used to increase an application's concurrency.
A thread represents a relatively independent set of instructions within a program. A thread is a control point within a process. It shares global resources within the context of the process (address space, open files, user credentials, quotas, etc). Threads also have private resources (program counter, stack, register context, etc).
The main benefit of threads (as compared to multiple processes) is that the context switches are much cheaper than those required to change current processes. Sun reports that a fork() takes 30 times as long as an unbound thread creation and 5 times as long as a bound thread creation.
Even within a single-processor environment, multiple threads are advantageous because one thread may be able to progress even though another thread is blocked while waiting for a resource.
Interprocess communication also takes considerably less time for threads than for processes, since global data can be shared instantly.
Kernel Threads
A kernel thread is the entity that is scheduled by the kernel. If no lightweight process is attached, it is also known as a system thread. It uses kernel text and global data, but has its own kernel stack, as well as a data structure to hold scheduling and syncronization information.
Kernel threads store the following in their data structure:
• Copy of the kernel registers.
• Priority and scheduling information.
• Pointers to put the thread on the scheduler or wait queue.
• Pointer to the stack.
• Pointers to associated LWP and proc structures.
• Pointers to maintain queues of threads in a process and threads in the system.
• Information about the associated LWP (as appropriate).
Kernel threads can be independently scheduled on CPUs. Context switching between kernel threads is very fast because memory mappings do not have to be flushed.
Lightweight Processes
A lightweight process can be considered as the swappable portion of a kernel thread.
Another way to look at a lightweight process is to think of them as "virtual CPUs" which perform the processing for applications. Application threads are attached to available lightweight processes, which are attached to a a kernel thread, which is scheduled on the system's CPU dispatch queue.
LWPs can make system calls and can block while waiting for resources. All LWPs in a process share a common address space. IPC (interprocess communication) facilities exist for coordinating access to shared resources.
LWPs contain the following information in their data structure:
• Saved values of user-level registers (if the LWP is not active)
• System call arguments, results, error codes.
• Signal handling information.
• Data for resource useage and profiling.
• Virtual time alarms.
• User time/CPU usage.
• Pointer to the associated kernel thread.
• Pointer to the associated proc structure.
By default, one LWP is assigned to each process; additional LWPs are created if all the process's LWPs are sleeping and there are additional user threads that libthread can schedule. The programmer can specify that threads are bound to LWPs.
Lightweight process information for a process can be examined with ps -elcL.
User Threads
User threads are scheduled on their LWPs via a scheduler in libthread. This scheduler does implement priorities, but does not implement time slicing. If time slicing is desired, it must be programmed in.
Locking issues must also be carefully considered by the programmer in order to prevent several threads from blocking on a single resource.
User threads are also responsible for handling of SIGSEGV (segmentation violation) signals, since the kernel does not keep track of user thread stacks.
Each thread has the following characteristics:
• Has its own stack.
• Shares the process address space.
• Executes independently (and perhaps concurrently with other threads).
• Completely invisible from outside the process.
• Cannot be controlled from the command line.
• No system protection between threads in a process; the programmer is responsible for interactions.
• Can share information between threads without IPC overhead.
Higher numbered priorities are given precedence. The scheduling page contains additional information on how priorities are set.
Zombie Processes
When a process dies, it becomes a zombie process. Normally, the parent performs a wait() and cleans up the PID. Sometimes, the parent receives too many SIGCHLD signals at once, but can only handle one at a time. It is possible to resend the signal on behalf of the child via kill -18 PPID. Killing the parent or rebooting will also clean up zombies. The correct answer is to fix the buggy parent code that failed to perform the wait() properly.
Aside from their inherent sloppiness, the only problem with zombies is that they take up a place in the process table.
Kernel Tunables
The following kernel tunables are important when looking at processes:
• maxusers: By default, this is set to 2 less than the number of Mb of physical memory, up to 1024. It can be set up to 2048 manually in the /etc/system file.
• max_nprocs: Maximum number of processes that can be active simultaneously on the system. The default for this is (16 x maxusers) + 10. The minimum setting for this is 138, the maximum is 30,000.
• maxuprc: The default setting for this is max_nprocs - 5. The minimum is 133, the maximum is . This is the numberof processes a single non-root user can create.
• ndquot: This is the number of disk quota structures. The default for this is (maxusers x 10) + max_nprocs. The minimum is 213.
• pt_cnt: Sets the number of System V ptys.
• npty: Sets the number of BSD ptys. (Should be set to pt_cnt.)
• sad_cnt: Sets the number of STREAMS addressable devices. (Should be set to 2 x pt_cnt.)
• nautopush: Sets the number of STREAMS autopush entries. (Should be set to pt_cnt.)
• ncsize: Sets DNLC size.
• ufs_ninode: Sets inode cache size.
proc Commands
The proc tools are useful for tracing attributes of processes. These utilities include:
• pflags: Prints the tracing flags, pending and held signals and other /proc status information for each LWP.
• pcred: Prints credentials (ie, EUID/EGID, RUID/EGID, saved UID/GIDs).
• pmap: Prints process address space map.
• pldd: Lists dynamic libraries linked to the process.
• psig: Lists signal actions.
• pstack: Prints a stack trace for each LWP in the process.
• pfiles: Reports fstat, fcntl information for all open files.
• pwdx: Prints each process's working directory.
• pstop: Stops process.
• prun: Starts stopped process.
• pwait: Wait for specified processes to terminate.
• ptree: Prints process tree for process.
• ptime: Times the command using microstate accounting; does not time children.
These commands can be run against a specific process, but most of them can also be run against all processes on the system. See the above- referenced man page for details.

Process Virtual Memory
Each process maps either 2^32 or 2^44 bytes of memory (depending on whether the OS is running in 32 or 64-bit mode). This works out to 4GB or 16TB. Not all of this memory is allocated (used); the virtual memory is used as address space that can be mapped to actual memory resources.
Virtual memory structure for a process can be described as in this diagram:


While this is part of the address space, it cannot be addressed directly by the process. It must be addressed via system calls.

Used by the program for variables and storage. It grows and shrinks in size depending on what routines are called and what their stack space requirements are. It is normally about 8MB in size.
Shared Libraries
Shared libraries are position independent so that they can be shared by all programs that want to use them. One common example is

This is the address space that is unallocated and unused. It does not tie up physical memory. For most processes, this is the largest portion of the virtual memory for the process.

Used for some types of working storage. It is allocated by the malloc function.

Uninitialized variables. These are not part of the executable file and their initial value is set to zeros.

Global variables, constants, static variables from the program.

Set of instructions from the compiler-generated executable file.
The virtual memory map for a process can be displayed using the pmap command.
Additional information is available on the Processes page.

Solaris Filesystem Troubleshooting
Filesystem corruption can be detected and often repaired by the format and fsck commands. If the filesystem corruption is not due to an improper system shutdown, the hard drive hardware may need to be replaced.
ufs filesystems contain the following types of blocks:
• boot block: This stores information used to boot the system.
• superblock: Much of the filesystems internal information is stored in these.
• inode: Stores location information about a file--everything except for the file name. The number of inodes in a filesystem can be changed from the default if newfs -i is used to create the filesystem.
• data block: The file's data is stored in these.
The fsck command is run on each filesystem at boot time. This utility checks the internal consistency of the filesystem, and can make simple repairs on its own. More complex repairs require feedback from the root user, either in terms of a "y" keyboard response to queries, or invocation with the -y option.
If fsck cannot determine where a file belongs, the file may be renamed to its inode number and placed in the filesystem's lost+found directory. If a file is missing after a noisy fsck session, it may still be intact in the lost+found directory.
Sometimes the fsck command complains that it cannot find the superblock. Alternative superblock locations were created by newfs at the time that the filesystem was created. The newfs -N command can be invoked to nondestructively discover the superblock locations for the filesystem.
ufs filesystems can carry "state flags" that have the value of fsclean, fsstable, fsactive or fsbad (unknown). These can be used by fsck during boot time to skip past filesystems that are believed to be okay.
The analyze option of format can be used to examine the hard drive for flaws in a nondestructive fashion.
df can be used to check a filesystem's available space. Of particular interest is df -kl, which checks available space for all local filesystems and prints out the statistics in kilobytes. Solaris 10 also allows us to use df -h, which presents the statistics in a more human-friendly form that doesn't require counting digits to decide whether a file is 100M or 1G in size.
du can be used to check space used by a directory. In particular, du -dsk will report useage in kilobytes of a directory and its descendants, without including space totals from other filesystems.
Filesystem Tuning
Filesystem performance can be improved by looking at filesystem caching issues.
The following tuning parameters may be valuable in tuning filesystem performance with tunefs or mkfs/newfs:
• inode count: The default is based upon an assumption of average file sizes of 2 KB. This can be set with mkfs/newfs at the time of filesystem creation.
• time/space optimization: Optimization can be set to allow for fastest performance or most efficient space useage.
• minfree: In Solaris 2.6+, this is set to (64 MB / filesystem size) x 100. Filesystems in earlier OS versions reserved 10%. This parameter specifies how much space is to be left empty in order to preserve filesystem performance.
• maxbpg: This is the maximum number of blocks a file can leave in a single cylinder group. Increasing this limit can improve large file performance, but may have a negative impact on small file performance.
Filesystem Performance Monitoring
McDougall, Mauro and Gregg suggest that the best way to see if I/O is a problem at all is to look at the amount of time spent on POSIX read() and write() system calls via DTrace. If so, we need to look at the raw disk I/O performance.

As with most of the monitoring commands, the first line of iostat reflects a summary of statistics since boot time. To look at meaningful real-time data, run iostat with a time step (eg iostat 30) and look at the lines that report summaries over the time step intervals.
For Solaris 2.6 and higher, use iostat -xPnce 30 to get information including the common device names of the disk partitions, CPU statistics, error statistics, and extended disk statistics.
For Solaris 2.5.1 and earlier, or for more compact output, use iostat -xc 30 to get the extended disk and CPU statistics.
In either case, the information reported is:
• disk: Disk device name.
• r/s, w/s: Average reads/writes per second.
• Kr/s, Kw/s: Average Kb read/written per second.
• wait: Time spent by a process while waiting for block
(eg disk) I/O to complete. (See Notes on Odd Behavior below.)
• actv: Number of active requests in the hardware queue.
• %w: Occupancy of the wait queue.
• %b: Occupancy of the active queue with the device busy.
• svc_t: Service time (ms). Includes everything: wait time, active queue time, seek rotation, transfer time.
• us/sy: User/system CPU time (%).
• wt: Wait for I/O (%).
• id: Idle time (%).
Notes on Odd Behavior
The "wait" time reported by iostat refers to time spent by a process while waiting for block device (such as disk) I/O to finish. In Solaris 2.6 and earlier, the calculation algorithm sometimes overstates the problem on multi-processor machines, since it does not take into account that an I/O wait on one CPU does not mean that I/O is blocked for processes on the other CPUs. Solaris 7 has corrected this problem.
iostat also sometimes reports excessive svc_t (service time) readings for disks that are very inactive. This is due to the action of fsflush keeping the data in memory and on the disk up-to-date. Since many writes are specified over a very short period of time to random parts of the disk, a queue forms briefly, and the average service time goes up. svc_t should only be taken seriously on a disk that is showing 5% or more activity.

mpstat reports information which is useful in understanding lock contention and CPU loading issues.
mpstat reports the following:
• CPU: Processor ID
• minf: Minor faults
• mjf: Major fault
• xcal: Processor cross-calls (when one CPU wakes up another by interrupting it). If this exceeds 200/second, the application in question may need to be examined.
• intr: Interrupts.
• ithr: Interrupts as threads (except clock).
• csw: Context switches
• icsw: Involuntary context switches (this is probably the more relevant statistic when examining performance issues.)
• migr: Thread migrations to another processor. If the migr measurement of mpstat is greater than 500, rechoose_interval should be sent longer in the kernel.
• smtx: Number of times a CPU failed to obtain a mutex.
• srw: Number of times a CPU failed to obtain a read/write lock on the first try.
• syscl: Number of system calls.
• usr/sys/wt/idl: User/system/wait/idle CPU percentages.

netstat provides useful information regarding traffic flow.
In particular, netstat -i lists statistics for each interface, netstat -s provides a full listing of several counters, and netstat -rs provides routing table statistics. netstat -an reports all open ports.
netstat -k provides a useful summary of several network-related statistics up through Solaris 9, but this option was removed in Solaris 10 in favor of the /bin/kstat command. Through Solaris 9, netstat -k provides a listing of several component kstat statistics.
Here are some of the issues that can be revealed with netstat:
• netstat -i: (Collis+Ierrs+Oerrs)/(Ipkts+Opkts) > 2%: This may indicate a network hardware issue.

• netstat -i: (Collis/Opkts) > 10%: The interface is overloaded. Traffic will need to be reduced or redistributed to other interfaces or servers.

• netstat -i: (Ierrs/Ipkts) > 25%: Packets are probably being dropped by the host, indicating an overloaded network (and/or server). Retransmissions can be dropped by reducing the rsize and wsize mount parameters to 2048 on the clients. Note that this is a temporary workaround, since this has the net effect of reducing maximum NFS throughput on the segment.

• netstat -s: If significant numbers of packets arrive with bad headers, bad data length or bad checksums, check the network hardware.

• netstat -i: If there are more than 120 collisions/second, the network is overloaded. See the suggestions above.

• netstat -i: If the sum of input and output packets is higher than about 600 for a 10Mbs interface or 6000 for a 100Mbs interface, the network segment is too busy. See the suggestions above.
• netstat -r: This form of the command provides the routing table. Make sure that the routes are as you expect them to be.

nfsstat can be used to examine NFS performance.
nfsstat -s reports server-side statistics. In particular, the following are important:
• calls: Total RPC calls received.
• badcalls: Total number of calls rejected by the RPC layer.
• nullrecv: Number of times an RPC call was not available even though it was believed to have been received.
• badlen: Number of RPC calls with a length shorter than that allowed for RPC calls.
• xdrcall: Number of RPC calls whose header could not be decoded by XDR (External Data Representation).
• readlink: Number of times a symbolic link was read.
• getattr: Number of attribute requests.
• null: Null calls are made by the automounter when looking for a server for a filesystem.
• writes: Data written to an exported filesystem.
Sun recommends the following tuning actions for some common conditions:
• writes > 10%: Write caching (either array-based or host-based, such as a Prestoserv card) would speed up operation.
• badcalls >> 0: The network may be overloaded and should be checked out. The rsize and wsize mount options can be set on the client side to reduce the effect of a noisy network, but this should only be considered a temporary workaround.
• readlink > 10%: Replace symbolic links with directories on the server.
• getattr > 40%: The client attribute cache can be increased by setting the actimeo mount option. Note that this is not appropriate where the attributes change frequently, such as on a mail spool. In these cases, mount the filesystems with the noac option.
nfsstat -c reports client-side statistics. The following statistics are of particular interest:
• calls: Total number of calls made.
• badcalls: Total number of calls rejected by RPC.
• retrans: Total number of retransmissions. If this number is larger than 5%, the requests are not reaching the server consistently. This may indicate a network or routing problem.
• badxid: Number of times a duplicate acknowledgement was received for a single request. If this number is roughly the same as badcalls, the network is congested. The rsize and wsize mount options can be set on the client side to reduce the effect of a noisy network, but this should only be considered a temporary workaround.
If on the other hand, badxid=0, this can be an indication of a slow network connection.
• timeout: Number of calls that timed out. If this is roughly equal to badxid, the requests are reaching the server, but the server is slow.
• wait: Number of times a call had to wait because a client handle was not available.
• newcred: Number of times the authentication was refreshed.
• null: A large number of null calls indicates that the automounter is retrying the mount frequently. The timeo parameter should be changed in the automounter configuration.
nfsstat -m (from the client) provides server-based performance data.
• srtt: Smoothed round-trip time. If this number is larger than 50ms, the mount point is slow.
• dev: Estimated deviation.
• cur: Current backed-off timeout value.
• Lookups: If cur>80 ms, the requests are taking too long.
• Reads: If cur>150 ms, the requests are taking too long.
• Writes: If cur>250 ms, the requests are taking too long.
In Unix, every object is either a file or a process. With the /proc virtual file system, even processes may be treated like files.
/proc (or procfs) is a virtual file system that allows us to examine processes like files. This means that /proc allows us to use file-like operations and intuitions when looking at processes. /proc does not occupy disk space; it is located in working memory. This structure was originally designed as a programming interface for writing debuggers, but it has grown considerably since then.
To avoid confusion, we will refer to the virtual file system as /proc or procfs. The man page for procfs is proc(4). proc, on the other hand, will be used to refer to the process data structure discussed in the Process Structure page.
Under /proc is a list of numbers, each of which is a Process ID (PID) for a process on our system. Under these directories are subdirectories referring to the different components of interest of each process. This directory structure can be examined directly, but we usually prefer to use commands written to extract information from this structure. These are known as the p-commands.
• pcred: Display process credentials (eg EUID/EGID, RUID/RGID, saved UIDs/GIDs)
• pfiles: Reports fstat() and fcntl() information for all open files. This includes information on the inode number, file system, ownership and size.
• pflags: Prints the tracing flags, pending and held signals and other /proc status information for each LWP.
• pgrep: Finds processes matching certain criteria.
• pkill: Kills specified processes.
• pldd: Lists dynamic libraries linked to the process.
• pmap: Prints process address space map.
• prun: Starts stopped processes.
• prstat: Display process performance-related statistics.
• ps: List process information.
• psig: Lists signal actions.
• pstack: Prints a stack trace for each LWP in the process.
• pstop: Stops the process.
• ptime: Times the command; does not time children.
• ptree: Prints process genealogy.
• pwait: Wait for specified processes to complete.
• pwdx: Prints process working directory.
prstat Example 1
CPU Saturation is can be directly measured via prstat. (Saturation refers to a situation where there is not enough CPU capacity to adequately handle requests for processing resources.) Saturation can be measured directly by looking at the CPU latency time for each thread reported by prstat -mL. (LAT is reported as a percentage of the time that a thread is waiting to use a processor.)
This example shows the prstat -mL output from a single-CPU system that has been overloaded. Notice the load average and LAT numbers.
2724 root 24 0.2 0.0 0.0 0.0 0.0 2.2 74 284 423 361 0 gzip/1
2729 root 21 0.3 0.0 0.0 0.0 0.0 3.3 75 396 564 518 0 gzip/1
2733 root 20 0.3 0.0 0.0 0.0 0.0 5.3 75 391 514 484 0 gzip/1
2737 root 14 0.2 0.0 0.0 0.0 0.0 4.1 81 176 415 383 0 gzip/1
2730 root 3.3 0.3 0.0 0.0 0.0 0.0 96 0.7 602 258 505 0 gunzip/1
2734 root 2.9 0.3 0.0 0.0 0.0 0.0 92 4.5 522 280 457 0 gunzip/1
2738 root 2.7 0.2 0.0 0.0 0.0 0.0 93 3.9 377 147 370 0 gunzip/1
2725 root 2.4 0.2 0.0 0.0 0.0 0.0 95 2.4 495 179 355 0 gunzip/1
2728 root 0.1 1.4 0.0 0.0 0.0 0.0 97 1.7 769 11 2K 0 tar/1
2732 root 0.1 1.3 0.0 0.0 0.0 0.0 99 0.2 762 14 2K 0 tar/1
2723 root 0.0 1.1 0.0 0.0 0.0 0.0 99 0.1 564 7 1K 0 tar/1
2731 root 0.3 0.4 0.0 0.0 0.0 0.0 98 1.2 754 3 1K 0 tar/1
2735 root 0.3 0.4 0.0 0.0 0.0 0.0 98 0.9 722 0 1K 0 tar/1
2736 root 0.0 0.6 0.0 0.0 0.0 0.0 99 0.0 341 2 1K 0 tar/1
2726 root 0.3 0.3 0.0 0.0 0.0 0.0 98 1.0 473 145 1K 0 tar/1
2739 root 0.2 0.2 0.0 0.0 0.0 0.0 99 0.3 335 1 664 0 tar/1
2749 scromar 0.0 0.1 0.0 0.0 0.0 0.0 100 0.0 23 0 194 0 prstat/1
337 root 0.0 0.0 0.0 0.0 0.0 0.0 100 0.0 6 0 36 6 xntpd/1
2716 scromar 0.0 0.0 0.0 0.0 0.0 0.0 100 0.0 3 1 21 0 sshd/1
124 root 0.0 0.0 0.0 0.0 0.0 0.0 100 0.0 3 0 17 0 picld/4
119 root 0.0 0.0 0.0 0.0 0.0 0.0 100 0.0 21 0 63 0 nscd/26
Total: 51 processes, 164 lwps, load averages: 4.12, 2.13, 0.88
prstat Example 2
In this case, we sort prstat output to look for the processes with heavy memory utilization:
# prstat -s rss
471 juser 125M 58M sleep 59 0 4:26:46 0.6% java/17
200 daemon 62M 55M sleep 59 0 0:01:21 0.0% nfsmapid/4
18296 juser 116M 39M sleep 26 11 0:05:36 0.1% java/23
254 root 3968K 1016K sleep 59 0 0:00:03 0.0% sshd/1
Total: 47 processes, 221 lwps, load averages: 0.20, 0.21, 0.20
Other Usage Examples
# ps -ef | grep more | grep -v grep
root 18494 8025 0 08:53:09 pts/3 0:00 more
# pgrep more
# pmap -x 18494
18494: more
Address Kbytes RSS Anon Locked Mode Mapped File
00010000 32 32 - - r-x-- more
00028000 8 8 8 - rwx-- more
0002A000 16 16 16 - rwx-- [ heap ]
FF200000 864 824 - - r-x--
FF2E8000 32 32 32 - rwx--
FF2F0000 8 8 8 - rwx--
FF300000 16 16 - - r-x--
FF312000 16 16 16 - rwx--
FF330000 8 8 - - r-x--
FF340000 8 8 8 - rwx-- [ anon ]
FF350000 168 104 - - r-x--
FF38A000 32 32 24 - rwx--
FF392000 8 8 8 - rwx--
FF3A0000 24 16 16 - rwx-- [ anon ]
FF3B0000 184 184 - - r-x--
FF3EE000 8 8 8 - rwx--
FF3F0000 8 8 8 - rwx--
FFBFC000 16 16 16 - rw--- [ stack ]
-------- ------- ------- ------- -------
total Kb 1456 1344 168 -
# pstack 18494
18494: more
ff2c0c7c read (2, ffbff697, 1)
00015684 ???????? (0, 1, 43858, ff369ad4, 0, 28b20)
000149a4 ???????? (ffbff82f, 28400, 15000000, 28af6, 0, 28498)
00013ad8 ???????? (0, 28b10, 28c00, 400b0, ff2a4a74, 0)
00012780 ???????? (2a078, ff393050, 0, 28b00, 2a077, 6b)
00011c68 main (28b10, ffffffff, 28c00, 0, 0, 1) + 684
000115cc _start (0, 0, 0, 0, 0, 0) + 108
# pfiles 18494
18494: more
Current rlimit: 256 file descriptors
0: S_IFIFO mode:0000 dev:292,0 ino:2083873 uid:0 gid:0 size:0
1: S_IFCHR mode:0620 dev:284,0 ino:12582922 uid:1000 gid:7 rdev:24,3
2: S_IFCHR mode:0620 dev:284,0 ino:12582922 uid:1000 gid:7 rdev:24,3
# pcred 18494
18494: e/r/suid=0 e/r/sgid=0
groups: 0 1 2 3 4 5 6 7 8 9 12
The word "sar" is used to refer to two related items:
1. The system activity report package
2. The system activity reporter
System Activity Report Package
This facility stores a great deal of performance data about a system. This information is invaluable when attempting to identify the source of a performance problem.
The Report Package can be enabled by uncommenting the appropriate lines in the sys crontab. The sa1 program stores performance data in the /var/adm/sa directory. sa2 writes reports from this data, and sadc is a more general version of sa1.
In practice, I do not find that the sa2-produced reports are terribly useful in most cases. Depending on the issue being examined, it may be sufficient to run sa1 at intervals that can be set in the sys crontab.
Alternatively, sar can be used on the command line to look at performance over different time slices or over a constricted period of time:

sar -A -o outfile 5 2000

(Here, "5" represents the time slice and "2000" represents the number of samples to be taken. "outfile" is the output file where the data will be stored.)
The data from this file can be read by using the "-f" option (see below).

System Activity Reporter
sar has several options that allow it to process the data collected by sa1 in different ways:
• -a: Reports file system access statistics. Can be used to look at issues related to the DNLC.

o iget/s: Rate of requests for inodes not in the DNLC. An iget will be issued for each path component of the file's path.

o namei/s: Rate of file system path searches. (If the directory name is not in the DNLC, iget calls are made.)

o dirbk/s: Rate of directory block reads.

• -A: Reports all data.

• -b: Buffer activity reporter:

o bread/s, bwrit/s: Transfer rates (per second) between system buffers and block devices (such as disks).

o lread/s, lwrit/s: System buffer access rates (per second).

o %rcache, %wcache: Cache hit rates (%).

o pread/s, pwrit/s: Transfer rates between system buffers and character devices.

• -c: System call reporter:

o scall/s: System call rate (per second).

o sread/s, swrit/s, fork/s, exec/s: Call rate for these calls (per second).

o rchar/s, wchar/s: Transfer rate (characters per second).

• -d: Disk activity (actually, block device activity):

o %busy: % of time servicing a transfer request.

o avque: Average number of outstanding requests.

o r+w/s: Rate of reads+writes (transfers per second).

o blks/s: Rate of 512-byte blocks transferred (per second).

o avwait: Average wait time (ms).

o avserv: Average service time (ms). (For block devices, this includes seek rotation and data transfer times. Note that the iostat svc_t is equivalent to the avwait+avserv.)

• -e HH:MM: CPU useage up to time specified.

• -f filename: Use filename as the source for the binary sar data. The default is to use today's file from /var/adm/sa.

• -g: Paging activity (see "Paging" for more details):

o pgout/s: Page-outs (requests per second).

o ppgout/s: Page-outs (pages per second).

o pgfree/s: Pages freed by the page scanner (pages per second).

o pgscan/s: Scan rate (pages per second).

o %ufs_ipf: Percentage of UFS inodes removed from the free list while still pointing at reuseable memory pages. This is the same as the percentage of igets that force page flushes.

• -i sec: Set the data collection interval to i seconds.

• -k: Kernel memory allocation:

o sml_mem: Amount of virtual memory available for the small pool (bytes). (Small requests are less than 256 bytes)

o lg_mem: Amount of virtual memory available for the large pool (bytes). (512 bytes-4 Kb)

o ovsz_alloc: Memory allocated to oversize requests (bytes). Oversize requests are dynamically allocated, so there is no pool. (Oversize requests are larger than 4 Kb)

o alloc: Amount of memory allocated to a pool (bytes). The total KMA useage is the sum of these columns.

o fail: Number of requests that failed.

• -m: Message and semaphore activities.

o msg/s, sema/s: Message and semaphore statistics (operations per second).

• -o filename: Saves output to filename.

• -p: Paging activities.

o atch/s: Attaches (per second). (This is the number of page faults that are filled by reclaiming a page already in memory.)

o pgin/s: Page-in requests (per second) to file systems.

o ppgin/s: Page-ins (per second). (Multiple pages may be affected by a single request.)

o pflt/s: Page faults from protection errors (per second).

o vflts/s: Address translation page faults (per second). (This happens when a valid page is not in memory. It is comparable to the vmstat-reported page/mf value.)

o slock/s: Faults caused by software lock requests that require physical I/O (per second).

• -q: Run queue length and percentage of the time that the run queue is occupied.

• -r: Unused memory pages and disk blocks.

o freemem: Pages available for use (Use pagesize to determine the size of the pages).

o freeswap: Disk blocks available in swap (512-byte blocks).

• -s time: Start looking at data from time onward.

• -u: CPU utilization.

o %usr: User time.

o %sys: System time.

o %wio: Waiting for I/O (does not include time when another process could be schedule to the CPU).

o %idle: Idle time.

• -v: Status of process, inode, file tables.

o proc-sz: Number of process entries (proc structures) currently in use, compared with max_nprocs.

o inod-sz: Number of inodes in memory compared with the number currently allocated in the kernel.

o file-sz: Number of entries in and size of the open file table in the kernel.

o lock-sz: Shared memory record table entries currently used/allocated in the kernel. This size is reported as 0 for standards compliance (space is allocated dynamically for this purpose).

o ov: Overflows between sampling points.

• -w: System swapping and switching activity.

o swpin/s, swpot/s, bswin/s, bswot/s: Number of LWP transfers or 512-byte blocks per second.

o pswch/s: Process switches (per second).

• -y: TTY device activity.

o rawch/s, canch/s, outch/s: Input character rate, character rate processed by canonical queue, output character rate.

o rcvin/s, xmtin/s, mdmin/s: Receive, transmit and modem interrupt rates.

The first line of vmstat represents a summary of information since boot time. To obtain useful real-time statistics, run vmstat with a time step (eg vmstat 30).
The vmstat output columns are as follows use the pagesize command to determine the size of the pages):
• procs or kthr/r: Run queue length.
• procs or kthr/b: Processes blocked while waiting for I/O.
• procs or kthr/w: Idle processes which have been swapped.
• memory/swap: Free, unreserved swap space (Kb).
• memory/free: Free memory (Kb). (Note that this will grow until it reaches lotsfree, at which point the page scanner is started. See "Paging" for more details.)
• page/re: Pages reclaimed from the free list. (If a page on the free list still contains data needed for a new request, it can be remapped.)
• page/mf: Minor faults (page in memory, but not mapped). (If the page is still in memory, a minor fault remaps the page. It is comparable to the vflts value reported by sar -p.)
• page/pi: Paged in from swap (Kb/s). (When a page is brought back from the swap device, the process will stop execution and wait. This may affect performance.)
• page/po: Paged out to swap (Kb/s). (The page has been written and freed. This can be the result of activity by the pageout scanner, a file close, or fsflush.)
• page/fr: Freed or destroyed (Kb/s). (This column reports the activity of the page scanner.)
• page/de: Freed after writes (Kb/s). (These pages have been freed due to a pageout.)
• page/sr: Scan rate (pages). Note that this number is not reported as a "rate," but as a total number of pages scanned.
• disk/s#: Disk activity for disk # (I/O's per second).
• faults/in: Interrupts (per second).
• faults/sy: System calls (per second).
• faults/cs: Context switches (per second).
• cpu/us: User CPU time (%).
• cpu/sy: Kernel CPU time (%).
• cpu/id: Idle + I/O wait CPU time (%).

vmstat -i reports on hardware interrupts.
vmstat -s provides a summary of memory statistics, including statistics related to the DNLC, inode and rnode caches.
vmstat -S reports on swap-related statistics such as:
• si: Swapped in (Kb/s).
• so: Swap outs (Kb/s).
(Note that the man page for vmstat -s incorrectly describes the swap queue length. In Solaris 2, the swap queue length is the number of idle swapped-out processes. (In SunOS 4, this referred to the number of active swapped-out processes.)
Solaris 8
vmstat under Solaris 8 will report different statistics than would be expected under an earlier version of Solaris due to a different paging algorithm:
• Page Reclaim rate higher.
• Higher reported Free Memory: A large component of the filesystem cache is reported as free memory.
• Low Scan Rates: Scan rates will be near zero unless there is a systemwide shortage of available memory.
vmstat -p reports paging activity details for applications (executables), data (anonymous) and filesystem activity.

System Configuration Files
For details about the files and commands summarized here, consult the appropriate man pages or

File Description
/etc/bootparams Contains information regarding network boot clients.
/etc/cron.d/cron.deny Allow access to crontab for users listed in this file. If the file does not exist, access is permitted for users not in the /etc/cron.d/cron.deny file.
/etc/defaultdomain NIS domain set by /etc/init.d/inetinit
/etc/default/cron Sets cron logging with the CRONLOG variable.
/etc/default/login Controls root logins via specification of the CONSOLE variable, as well as variables for login logging thresholds and password requirements.
/etc/default/su Determines logging activity for su attempts via the SULOG and SYSLOG variables, sets some initial environment variables for su sessions.
/etc/dfs/dfstab Determines which directories will be NFS-shared at boot time. Each line is a share command.
/etc/dfs/sharetab Contains a table of resources that have been shared via share.
/etc/group Provides groupname translation information.
/etc/hostname.interface Assigns a hostname to interface; assigns an IP address by cross- referencing /etc/inet/hosts.
/etc/hosts.deny Determine which hosts will be allowed access to TCP wrapper mediated services.
/etc/hosts.equiv Determines which set of hosts will not need to provide passwords when using the "r" remote access commands (eg rlogin, rsh, rexec)
/etc/hosts Associates hostnames and IP addresses.
/etc/inetd.conf Identifies the services that are started by inetd as well as the manner in which they are started. inetd.conf may even specify that TCP wrappers be used to protect a service.

/etc/inittab inittab is used by init to determine scripts to for different run levels as well as a default run level.
/etc/logindevperm Contains information to change permissions for devices upon console logins.
/etc/magic Database of magic numbers that identify file types for file.
/etc/aliases Contains mail aliases recognized by sendmail.
/etc/ Mail configuration file for sendmail.
/etc/minor_perm Specifies permissions for device files; used by drvconfig
/etc/mnttab Contains information about currently mounted resources.
/etc/name_to_major List of currently configured major device numbers; used by drvconfig.
/etc/netconfig Network configuration database read during network initialization.
/etc/netgroup Defines groups of hosts and/or users.
/etc/netmasks Determines default netmask settings.
/etc/nsswitch.conf Determines order in which different information sources are accessed when performing lookups.
/etc/path_to_inst Contents of physical device tree using physical device names and instance numbers.
/etc/protocols Known protocols.
/etc/remote Attributes for tip sessions.

/etc/rmtab Currently mounted filesystems.
/etc/rpc Available RPC programs.
/etc/services Well-known networking services and associated port numbers.
/etc/syslog.conf Configures syslogd logging.
/etc/system Can be used to force kernel module loading or set kernel tuneable parameters.

/etc/vfstab Information for mounting local and remote filesystems.
/var/adm/messages Main log file used by syslogd.
/var/adm/sulog Default log for recording use of su command.
/var/adm/utmpx User and accounting information.
/var/adm/wtmpx User login and accounting information.
/var/local/etc/ftpusers wu-ftpd configuration files to set ftp access rights, conversion/compression types, and a list of userids to exclude from ftp operations.
/var/lp/log Print services activity log.
/var/sadm/install/contents Database of installed software packages.
/var/saf/_log Logs activity of SAF (Service Access Facility).

Important Header Files
Some of these header files contain important information that can be used when analyzing a core dump or setting parameter values.
Location Description
/usr/include/sys Most system header files.
/usr/include/vm Header files describing virtual memory structures.
/usr/include/sys/fs Header files describing file system structures and types.
/usr/platform/`arch -k`/include/sys Architecture dependent structures.
/usr/platform/`arch -k`/include/vm Architecture-dependent virtual memory structures.
/usr/include/[net, nfs, rpc, protocols, inet, netinet] Define networking data structures.

The /etc/inittab File
The /etc/inittab file plays a crucial role in the boot sequence.
For versions of Solaris prior to version 10, the /etc/inittab was edited manually. Solaris 10+ manages the /etc/inittab through SMF. The Solaris 10 inittab should not be edited directly
The default Solaris 10 inittab contains the following:
1. ap::sysinit:/sbin/autopush -f /etc/iu.ap
2. sp::sysinit:/sbin/soconfig -f /etc/sock2path
3. smf::sysinit:/lib/svc/bin/svc.startd >/dev/msglog 2<>/dev/msglog
4. p3:s1234:powerfail:/usr/sbin/shutdown -y -i5 -g0 >/dev/msglog 2<>/dev/...
The lines accomplish the following:
1. Initializes Streams
2. Configures socket transport providers
3. Initializes SMF master restarter
4. Describes a power fail shutdown
In particular, the default keyword is not used any more in Solaris 10. Instead, the default run level is determined within the SMF profile.
When the init process is started, it first sets environment variables set in the /etc/default/init file; by default, only TIMEZONE is set. Then init executes process entries from the inittab that have sysinit set, and transfers control of the startup process to svc.startd.
Solaris 8 and 9
The line entries in the inittab file have the following format:
Here the id is a two-character unique identifier, runlevel indicates the run level involved, action indicates how the process is to be run, and process is the command to be executed.
At boot time, all entries with runlevel "sysinit" are run. Once these processes are run, the system moves towards the init level indicated by the "initdefault" line. For a default inittab, the line is:
(This indicates a default runlevel of 3.)
By default, the first script run from the inittab file is /etc/bcheckrc, which checks the state of the root and /usr filesystems. The line controlling this script has the following form:
fs::sysinit:/sbin/bcheckrc >/dev/console 2>&1 The inittab also controls what happens at each runlevel. For example, the default entry for runlevel 2 is:
s2:23:wait:/sbin/rc2 >/dev/console 2>&1 The action field of each entry will contain one of the following keywords:
• powerfail: The system has received a "powerfail" signal.
• wait: Wait for the command to be completed before proceeding.
• respawn: Restart the command.


  1. Hi this is Kathiresan i am having 3 years of experience as a dot net developer and i am certified. i have knowledge on OOPS concepts in .NET but dont know indepth. After learning android will be enough to get a good career in IT with good package? and i crossed Android Training in Chennai website where someone please help me to identity the syllabus covers everything or not??


  2. Awesome Blogs share more information and refer the link and refer the link Android Training in Chennai

  3. There are lots of information about latest technology and how to get trained in them, like Hadoop Training Chennai have spread around the web, but this is a unique one according to me. The strategy you have updated here will make me to get trained in future technologies(Hadoop Training in Chennai). By the way you are running a great blog. Thanks for sharing this (Salesforce Training in Chennai).

  4. I have read your blog, it was good to read & I am getting some useful info's through your blog keep sharing... Informatica is an ETL tools helps to transform your old business leads into new vision. Learn Informatica training in chennai from corporate professionals with very good experience in informatica tool.
    Best Informatica Training In Chennai|Informatica training center in Chennai|Informatica training chennai

  5. I am reading your post from the beginning, it was so interesting to read & I feel thanks to you for posting such a good blog, keep updates regularly.

    sas training in Chennai|sas course in Chennai|sas courses in chennai

  6. Very nice I gathered good information from this content.
    db2 training in chennai

  7. I really appreciate information shared above. It’s of great help. If someone want to learn Online (Virtual) instructor lead live training in Oracle Solaris, kindly contact us
    MaxMunus Offer World Class Virtual Instructor led training on Oracle Solaris. We have industry expert trainer. We provide Training Material and Software Support. MaxMunus has successfully conducted 100000+ trainings in India, USA, UK, Australlia, Switzerland, Qatar, Saudi Arabia, Bangladesh, Bahrain and UAE etc.
    For Demo Contact us:
    Name : Arunkumar U
    Email :
    Skype id: training_maxmunus
    Contact No.-+91-9738507310
    Company Website –

  8. Thanks for the awesome share
    We at Colan Infotech Private Limited best web design company in chennai,is Situated in US and India,
    will provide you best service in qa testing services
    Design Services and Colan Infotech has a group of exceedingly dedicated, inventive and creative experts with an energy for delivering exciting ,
    helpful and stylish Web and Mobile Applications, We are one of the best software testing services company
    and of course we stepped in bangalore too we are best qa and testing services
    can provide quality assurance and testing services,
    we are the best among the software testing company india

  9. Thanks for sharing this information article
    Hi we at Colan Infotech Private Limited , a company which is Situated in US and India, will provide you best service and our talented team will assure you best result and we are familiar with international markets, We work with customers in a wide variety of sectors. Our talented team can handle all the aspects of custom application development, we are the best among the dot net development companies in Chennai .
    asp .net web development company.
    We have quite an extensive experience working with
    asp .net development services. we are the only web development company
    which offer custom services to a wide range of industries by exceeding our client’s expectations.You can even interact directly with the team regarding your project, just as you would with your in-house team.
    hire programmers
    to achieve your dream product.
    Custom application development company, development companies, asp .net web development company,Hire asp .net programmers, web development services,dot net development companies in chennai
    Hire asp .net programmers.
    Here is a good resource if anyone in need of web development services
    dot net development companies in chennai.

  10. Thanks for the awesome share
    We at Colan Infotech Private Limited best web design company in chennai,is Situated in US and India,
    will provide you best service in qa testing services
    Design Services and Colan Infotech has a group of exceedingly dedicated, inventive and creative experts with an energy for delivering exciting ,
    helpful and stylish Web and Mobile Applications, We are one of the best software testing services company
    and of course we stepped in bangalore too we are best qa and testing services
    can provide quality assurance and testing services,
    we are the best among the software testing company india

  11. Thank you for sharing such a nice and interesting blog with us. i have seen that all will say the same thing repeatedly. But in your blog, I had a chance to get some useful and unique information
    We at Colan Infotech Private Limited best web design company in chennai,is Situated in US and India, will provide you best service in
    Professional Website Design Services and Colan Infotech has a group of exceedingly dedicated,
    inventive and creative experts with an energy for delivering exciting , helpful and stylish Web and Mobile Applications,
    We provide all sort of web designing services in chennai and
    of course we stepped in bangalore too we are best website designers in bangalore can provide web design services in bangalore, hire web designer india

  12. Thank you a lot for providing individuals with a very spectacular possibility to read critical reviews from this site.
    Best Java Training Institute Chennai

    Java Training Institute Bangalore

  13. Thanks a lot very much for the high quality and results-oriented help. I won’t think twice to endorse your blog post to anybody who wants and needs support about this area.
    selenium training in chennai

  14. Ciitnoida provides Core and java training institute in

    . We have a team of experienced Java professionals who help our students learn Java with the help of Live Base Projects. The object-

    oriented, java training in noida , class-based build

    of Java has made it one of most popular programming languages and the demand of professionals with certification in Advance Java training is at an

    all-time high not just in India but foreign countries too.

    By helping our students understand the fundamentals and Advance concepts of Java, we prepare them for a successful programming career. With over 13

    years of sound experience, we have successfully trained hundreds of students in Noida and have been able to turn ourselves into an institute for best

    Java training in Noida.

    java training institute in noida
    java training in noida
    best java training institute in noida
    java coaching in noida
    java institute in noida

  15. Sap Training Institute in Noida

    CIIT Noida provides Best SAP Training in Noida based on current industry standards that helps attendees to secure placements in their dream jobs at MNCs. CIIT Provides Best ERP SAP Training in Noida. CIIT is one of the most credible ERP SAP training institutes in Noida offering hands on practical knowledge and full job assistance with basic as well as advanced level ERP SAP training courses. At CIIT ERP SAP training in noida is conducted by subject specialist corporate professionals with 7+ years of experience in managing real-time ERP SAP projects. CIIT implements a blend of aERPemic learning and practical sessions to give the student optimum exposure that aids in the transformation of naïve students into thorough professionals that are easily recruited within the industry.

    At CIIT’s well-equipped ERP SAP training center in Noida aspirants learn the skills for ERP SAP Basis, ERP SAP ABAP, ERP SAP APO, ERP SAP Business Intelligence (BI), ERP SAP FICO, ERP SAP HANA, ERP SAP Production Planning, ERP SAP Supply Chain Management, ERP SAP Supplier Relationship Management, ERP SAP Training on real time projects along with ERP SAP placement training. ERP SAP Training in Noida has been designed as per latest industry trends and keeping in mind the advanced ERP SAP course content and syllabus based on the professional requirement of the student; helping them to get placement in Multinational companies and achieve their career goals.

  16. BCA Colleges in Noida

    CIIT Noida provides Sofracle Specialized B Tech colleges in Noida based on current industry standards that helps students to secure placements in their dream jobs at MNCs. CIIT provides Best B.Tech Training in Noida. It is one of the most trusted B.Tech course training institutes in Noida offering hands on practical knowledge and complete job assistance with basic as well as advanced B.Tech classes. CIITN is the best B.Tech college in Noida, greater noida, ghaziabad, delhi, gurgaon regoin .

    At CIIT’s well-equipped Sofracle Specialized M Tech colleges in Noida aspirants learn the skills for designing, analysis, manufacturing, research, sales, management, consulting and many more. At CIIT B.Tech student will do practical on real time projects along with the job placement and training. CIIT Sofracle Specialized M.Tech Classes in Noida has been designed as per latest IT industry trends and keeping in mind the advanced B.Tech course content and syllabus based on the professional requirement of the student; helping them to get placement in Multinational companies (MNCs) and achieve their career goals.

    MCA colleges in Noida we have high tech infrastructure and lab facilities and the options of choosing multiple job oriented courses after 12th at Noida Location. CIIT in Noida prepares thousands of engineers at reasonable B.Tech course fees keeping in mind training and B.Tech course duration and subjects requirement of each attendee.

    Engineering College in Noida"