Tag Archives: memory

Vmware NSX for vSphere 6.2.2 bugs — lost network connection

We are using VMware NSX in our production environment for a long time.
And recently we got some problem with NSX, the symptoms is

Some VMs will lose network connection after migrated to another VM;
New firewall rules are not able to apply on some of the VMs.

After engaged VMware, VMware confirmed that it’s a bug in NSX.

VMware assigned about 1.6G heap memory for NSX firewall on each of the ESX hosts. If you applied too much rules or you have too many VMs and you’ll reach the memory limit. Then you’ll get this issue…

Current fix is to upgrade to 6.2.3…

Reading a memory.dmp or other .dmp file

This can be accomplished with 7 easy steps:

Step 1. Obtain and install the debugging tools.

Debugging Tools Windows

All you need to install is the “Install Debugging Tools for Windows as a Standalone Component (from Windows SDK)” and during the install only select “Debugging Tools for Windows”. Everything else is used for more advanced troubleshooting or development, and isn’t needed here. Today I followed the link to “Install Debugging Tools for Windows as a Standalone Component (from Windows SDK)” although for a different OS you may need to follow a different link.

Step 2. From an elevated command prompt navigate to the debugging folder. For me with the latest tools on Windows Server 2012 it was at C:\Program Files (x86)\Windows Kits\8.1\Debuggers\x64\. You can specify the path during the install.

Step 3. Type the following:

kd –z C:\Windows\memory.dmp (or the path to your .dmp file)

Step 4. Type the following:

.logopen c:\debuglog.txt

Step 5. Type the following:

.sympath srv*c:\symbols*http://msdl.microsoft.com/download/symbols

If you computer can’t connect to internet, you can download the symbols from below link:

https://developer.microsoft.com/en-us/windows/hardware/download-symbols

Step 6. Type the following:

.reload;!analyze -v;r;kv;lmnt;.logclose;q

Step 7. Review the results by opening c:\debuglog.txt in your favorite text editor. Searching for PROCESS_NAME: will show which process had the fault. You can use the process name and other information from the dump to find clues and find answers in a web search. Usually the fault is with a hardware drivers of some sort, but there are many things that can cause crashes so the actual analyzing of the dump may take some research.

Often times a driver update will fix the issue. If the summary information doesn’t offer enough information then you’ll need to dig further into the debugging tools or open a CSS case with Microsoft. The steps above will provide you with a summary mostly-human-readable report from the dump. There is much more information available in the memory dump although it gets exponentially more difficult to track down the details the further you get into windows debugging.

Hopefully these quick steps are helpful for you as you troubleshoot the unwelcome BSOD.

Understanding Citrix Performance Issues

Bottleneck: provisioning services. Customers note there is excessive Network I/O and CPU utilization.
Bottleneck: vDisk fragmentation or server virtual instances. Customer notes there is excessive page file utilization and disk I/O.
Bottleneck: delays mounting new vDisks. Check for excessive Network and Disk I/O on delivery controllers.
Bottleneck: delivery controllers. Check for excessive historical CPU utilization.
Bottleneck: slow application enumeration. Check for excessive disk and network I/O on the data collectors.
Bottleneck: slow session creation noted within the director console: Check for historical CPU and Memoyr consumption, consider adding VCPU and memory when/where needed.
Bottleneck: higher than expected user logons. Check for high CPU and/or network utilization (not historical but may trend at random intervals). Add processing or new delivery controller if necessary to handle the expected loads.
Bottleneck: issues with local host cache (LHC). Disk and Page File I/O in excess can cause unanticipated issues with LHC. Alert and adjust when/where needed.
Bottleneck: Processor intensive apps. Check questionable servers for larger disk I/O and page file utilization. Consider adding more VCPU’s and/or memory to offset the demand on disk and page file.
Bottleneck: vDisk and/or Provisioning Services. Check for higher than normal CPU and/or Memory consumption as a deficiency will slow down the loading of vDisks and caching via Provisioning Services (PVS).
Bottleneck: Web interface authentication. Consider adding more memory and looking at network utilization trends. It may be necessary to either add more memory or to add an additional WI to your GSLB URL.
Bottleneck: slow PXE and vDisk. Check for memory and/or network utilization and consider addresssing depending on noted trends.
Bottleneck: target device latency. Check CPU and network I/O for spikes and/or trending issues.