- Generate vCenter Server and ESXi Log Bundles
On the VI client, highlight the vCenter and select File>Export>Export System Logs…
Under the vCenter Inventory tree select the logs you wish to export
Select the system logs you wish to export. It’s a good idea to only select the logs in the area you’re looking to troubleshoot and expand if required
Select your local download location and then review summary and click Finish, this process may take some time depending on the size of your enviroment
- Use esxcli system syslog to configure centralized logging on ESXi hosts
First check the the hosts firewall port is open to allow SYSLOGS to be sent to a remote host. On the hosts configuration tab select security profile then the properties of the firewall, ensure the syslog option is ticked (it is NOT by default)
SSH into the host you wish to configure remote syslog collection (when configuring multiple hosts you may wish to use the vMA or create a script)
Type the following command to show you the availble name spaces & commands under the esxcli system syslog syntax
esxcli system syslog
As you can see from the commands listed the availble namespace is config, the commands are mark & reload. It’s worth noting the reload command at this stage as it specifies the following
“Reload the log daemon to apply any new configuration options”
Running esxcli system syslog config will return the availble namespaces and commands within the config namespace
By running the first command above with the get command , we can run the following to show the current configuration values
esxcli system syslog config get
To set the configuration we need to use the esxcli system syslog config syntax with the Set command. To find the availble command options within the Set command we can run the following command
esxcli system syslog config set –help
Use the desired command options to configure remote syslog collection. The example below sets the syslog server & set a unique log directory.
esxcli system syslog config set –loghost=192.168.0.2 –logdir-unique=true
esxcli system syslog reload
You can then run the get command to ensure the settings have now taken effect
esxcli system syslog config get
If you have installed the syslog collector on your vCenter then you will be able to see the hosts listed under the Network Syslog Collector option in vCenter, however this does not dynamically update and you will have to close & re-open the VI-Client to show newly configured Hosts
If you cant see the Network Syslog Collector icon, ensure the Plug-in is properly installed. Details of how to install the VMware syslog collector are detailed below in the Install and Configure Vmware syslog Collector section.
- Test centralised logging configuration
If you have the VMware syslog collector installed on a Windows Based vCenter, then sys log files will be written to C:\ProgramData\Vmware\VMware syslog collector\data\ by default.
If you have enabled the –logdir-unique=true option when configuring remote syslog collection on the hosts then a sub folder will be created in that directory with the ESXi Hosts name.
Inside your unique folder you will see the syslog.log file.
To test that hosts are writting to the syslog file you can run the following esxcli command from the host.
esxcli system syslog mark -s “I’m a working syslog”
The test “I’m a working syslog” can be anything you want, it will be written to the syslog file.
We can then open the syslog.log file of the host we have just run the ESXCLI command on, and run a search for the text entry (in this case “I’m a working syslog”)
This is useful when running losts of commands on the host. You can simply put an entry into the syslog file of say esxcli system syslog mark -s “I started working here”
Then when troubleshooting problems, rather than search the whole syslog file to find the point you started working you can simple search for the text “I started working here” & you’ll know any errors after that will probably be of your own doing 🙂
- Analyze log entries to obtain configuration information
My advice would be, if you’re setting up a lab make such syslog collection is one of the first things you configure, you can then analyze the log files to see how configuration changes you make are reflected in the log files.
- Analyze log entries to identify and resolve issues
This is a skill that can only be learnt by real troubleshooting, my advise would be to know where to find the required log files to troubleshoot vSphere & SSO
- Install and configure VMware syslog Collector and ESXi Dump Collector
The syslog collector and the ESXi Dump Collector are both installed using the vCenter media.
Installation is self-explantory and is very much a Next>Next>Finish job. Once you have installed both of the collectors you will be able to see them within the vSphere Client.
VMware ESXi Dump Collector
VMware Syslog Collector
Details of how to configure hosts to log to a syslog collector and how to test are detailed in the objectives above.
To configure hosts to write to the ESXi Dump Collector you will need to do the following.
SSH to a host and run the following command
esxcli system coredump
As with the syslog configuration, this lists the availble Namespaces availble
As detailed we can configure the dump files to write to a VMFS file, Network or Partition. In this case we want to write to a network location so we will run the esxcli system coredump network command as with the syslog configuration we can run the following to pull the current configuration
esxcli system coredump network get
We can then run the following to get a list of command options included in the set command
esxcli system coredump network set –help
Please note the -e|–enable command which lists that it cannot be specified when setting the dump parameters below. In which case we will need to run this after we have set the new configuration.
You will need to know the Port number of the dump collector, you can find this within the Vi-Client Dump Collector option (screen shot earlier in this objective) the default port is 6500.
You will also need to select a VMK adapter that is on a subnet that is routable to the dump collector, I’d imagine in all scenerios this would be the MGMT VMK.
esxcli system coredump network set -v vmk0 -i 192.168.0.2 -o 6500
To then enable the configuration you will need to run
esxcli system coredump network set -e true
You can check the configuration changes have been made by running the get command
esxcli system coredump network get
Once this is configured, all Kernal Panic logs will be dumped on the coredump collector.
You can force a Kernel panic by running the following commands on a host… MAKE SURE IT IS NOT IN PRODUCTION!!!!
- Connect to the vSphere host via SSH.
- Type vsish
- Type set /reliability/crashMe/Panic
Once you get the PSOD you should see it dumping to the configured dump collector
You can then find the dump in the default location on the vCenter
C:\ProgramData\VMware\VMware ESXi Dump Collector\Data
There will be subfolders that relate to the name of your host.