Objective 5.4 – Configure, Manage, and analyze vSphere and SSO log files

  • Generate vCenter Server and ESXi Log Bundles

On the VI client, highlight the vCenter and select File>Export>Export System Logs…

logs

Under the vCenter Inventory tree select the logs you wish to export

logs1

Select the system logs you wish to export. It’s a good idea to only select the logs in the area you’re looking to troubleshoot and expand if required

logs2

Select your local download location and then review summary and click Finish, this process may take some time depending on the size of your enviroment

logs3

  • Use esxcli system syslog to configure centralized logging on ESXi hosts

First check the the hosts firewall port is open to allow SYSLOGS to be sent to a remote host. On the hosts configuration tab select security profile then the properties of the firewall, ensure the syslog option is ticked (it is NOT by default)

logs4

SSH into the host you wish to configure remote syslog collection (when configuring multiple hosts you may wish to use the vMA or create a script)

Type the following command to show you the availble name spaces & commands under the esxcli system syslog syntax

esxcli system syslog

esxcli

As you can see from the commands listed the availble namespace is config, the commands are mark & reload. It’s worth noting the reload command at this stage as it specifies the following

“Reload the log daemon to apply any new configuration options”

Running esxcli system syslog config will return the availble namespaces and commands within the config namespace

escxcli2

By running the first command above with the get command , we can run the following to show the current configuration values

esxcli system syslog config get

esxcli1

To set the configuration we need to use the esxcli system syslog config syntax with the Set command. To find the availble command options within the Set command we can run the following command

esxcli system syslog config set –help

esxcli3

Use the desired command options to configure remote syslog collection. The example below sets the syslog server & set a unique log directory.

esxcli system syslog config set –loghost=192.168.0.2 –logdir-unique=true

esxcli4
To active the new configuration you will need to run the reload command as noted above.

esxcli system syslog reload

You can then run the get command to ensure the settings have now taken effect

esxcli system syslog config get

esxcli5

If you have installed the syslog collector on your vCenter then you will be able to see the hosts listed under the Network Syslog Collector option in vCenter, however this does not dynamically update and you will have to close & re-open the VI-Client to show newly configured Hosts

If you cant see the Network Syslog Collector icon, ensure the Plug-in is properly installed. Details of how to install the VMware syslog collector are detailed below in the Install and Configure Vmware syslog Collector section.

esxcli6
esxcli7

  • Test centralised logging configuration

If you have the VMware syslog collector installed on a Windows Based vCenter, then sys log files will be written to C:\ProgramData\Vmware\VMware syslog collector\data\ by default.

If you have enabled the –logdir-unique=true option when configuring remote syslog collection on the hosts then a sub folder will be created in that directory with the ESXi Hosts name.

syslog

Inside your unique folder you will see the syslog.log file.

To test that hosts are writting to the syslog file you can run the following esxcli command from the host.

esxcli system syslog mark -s “I’m a working syslog”

The test “I’m a working syslog” can be anything you want, it will be written to the syslog file.

We can then open the syslog.log file of the host we have just run the ESXCLI command on, and run a search for the text entry (in this case “I’m a working syslog”)

syslog2

This is useful when running losts of commands on the host. You can simply put an entry into the syslog file of say esxcli system syslog mark -s “I started working here”

Then when troubleshooting problems, rather than search the whole syslog file to find the point you started working you can simple search for the text “I started working here” & you’ll know any errors after that will probably be of your own doing 🙂

  • Analyze log entries to obtain configuration information

My advice would be, if you’re setting up a lab make such syslog collection is one of the first things you configure, you can then analyze the log files to see how configuration changes you make are reflected in the log files.

  • Analyze log entries to identify and resolve issues

This is a skill that can only be learnt by real troubleshooting, my advise would be to know where to find the required log files to troubleshoot vSphere & SSO

  • Install and configure VMware syslog Collector and ESXi Dump Collector

The syslog collector and the ESXi Dump Collector are both installed using the vCenter media.

dump

Installation is self-explantory and is very much a Next>Next>Finish job.  Once you have installed both of the collectors you will be able to see them within the vSphere Client.

dump2

VMware ESXi Dump Collector

dump3

VMware Syslog Collector

dump4

Details of how to configure hosts to log to a syslog collector and how to test are detailed in the objectives above.

To configure hosts to write to the ESXi Dump Collector you will need to do the following.

SSH to a host and run the following command

esxcli system coredump

As with the syslog configuration, this lists the availble Namespaces availble

dump5

As detailed we can configure the dump files to write to a VMFS file, Network or Partition. In this case we want to write to a network location so we will run the esxcli system coredump network command as with the syslog configuration we can run the following to pull the current configuration

esxcli system coredump network get

We can then run the following to get a list of command options included in the set command

esxcli system coredump network set –help

dump6

Please note the -e|–enable command which lists that it cannot be specified when setting the dump parameters below. In which case we will need to run this after we have set the new configuration.

You will need to know the Port number of the dump collector, you can find this within the Vi-Client Dump Collector option (screen shot earlier in this objective) the default port is 6500.

You will also need to select a VMK adapter that is on a subnet that is routable to the dump collector, I’d imagine in all scenerios this would be the MGMT VMK.

esxcli system coredump network set -v vmk0 -i 192.168.0.2 -o 6500

dump7

To then enable the configuration you will need to run

esxcli system coredump network set -e true

You can check the configuration changes have been made by running the get command

esxcli system coredump network get

dump8

Once this is configured, all Kernal Panic logs will be dumped on the coredump collector.

You can force a Kernel panic by running the following commands on a host… MAKE SURE IT IS NOT IN PRODUCTION!!!!

  • Connect to the vSphere host via SSH.
  • Type vsish
  • Type set /reliability/crashMe/Panic

Once you get the PSOD you should see it dumping to the configured dump collector

CD

You can then find the dump in the default location on the vCenter

C:\ProgramData\VMware\VMware ESXi Dump Collector\Data

There will be subfolders that relate to the name of your host.

dumpfile

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s