heapdump定位内存泄露
工具下载hihttps://download.****.net/download/momo459548255/12832315
Linux下分析一个core内存泄露的过程（3s大法）
对于dump很大的原因分析
- 准备环境
找到对应dms.core的jenkins上打包出来的版本程序解压到和core一起的路径
/log/core/core/DMS
- 准备工具
把heapdump工具也拷贝到/log/core/core/DMS下面
- Gdb调试
gdb -c DMS_12409-1398407860-6.core DMS
bt
可以清晰的看到是new出异常，而且core这么大，基本可以断定是内存泄露了或者内存来不及释放.
- 内存泄露windows下当然用3s大法喽，linux下要用heapdump哦！！
./heapdump –h 帮助先看下
./heapdump -p DMS_12409-1398407860-6.core 准备调试环境，要等待一下
./heapdump -a >a_dms.txt 打印所有堆栈信息 !heap –s (windows)
./heapdump --stat >s_dms.txt 列出堆块统计信息 !heap –stat –h(windows)
cat s_dms.txt 可以清晰的看到前面两个占用最大22560*9373,53c0*9373；这里
我们可以知道这两块内存肯定是相关的，因为个数一样多; 很可能是一个结构体或者类里的两块buf，也可能是一个类或者结构体包含一buf；
cat a_dms.txt | grep 22560从整个堆栈中查找谁使用了这个22560大小的内存(下面部分)
哈哈差不多了，只要查出来0xffca7048是个什么东东就行了，查看虚函数表
(gdb) info symbol * 0xffca7048//注意此功能GDB7.6 版本才支持 GDB7.0版本不支持
!heap -flt s 22560(windows下可以看到所有这个大小对应的虚表)
No symbol matches *0xffca7048----伤心难过绝望啊，怎么就没有呢,那它就不是类喽;
!heapdump –f 0xffca7048 幸亏张博还提供了一个查找功能，可以逆向追溯，查找是谁引用了
可以看到0xffb22580引用了这个地址，注意上面的打印解释可以参考张博的案例解析
(gdb) info symbol *0xffb22580 如果你找不到的话，再./heapdump –f 再回溯下找
哈哈终于有了,咦？好像忘了还有一个53c0*9373，额?
省去上面的步骤，看个结果吧；
(gdb) info symbol *0x08f22580
又是这个类，这好像就验证了刚刚的说法，两个肯定相关，且53c0是类大小，22560是类里面的一个new出来的内存，可以看看结果哦
哈哈果然如此；
基本可以断定是这个类没有释放掉，导致类的_data也没有释放，看看代码就行了
2、3s解决不了的问题
昨天碰到个客户端1.6G的dump.使用3S大法查不出来。
这里写一个3S大法查不出来的内存泄露的排查思路。
第一步!heap –s
0:000> !heap -s
LFH Key : 0x27c8f1dd
Termination on corruption : DISABLED
Heap Flags Reserv Commit Virt Free List UCR Virt Lock Fast
(k) (k) (k) (k) length blocks cont. heap
-----------------------------------------------------------------------------
Virtual block: 27d30000 - 27d30000 (size 00000000)
Virtual block: 287f0000 - 287f0000 (size 00000000)
Virtual block: 28ba0000 - 28ba0000 (size 00000000)
002e0000 00000002 64960 56900 64960 131 458 8 3 0 LFH
00010000 00008000 64 4 64 2 1 1 0 0
Virtual block: 5ad70000 - 5ad70000 (size 00000000)
Virtual block: 5af80000 - 5af80000 (size 00000000)
Virtual block: 5cc90000 - 5cc90000 (size 00000000)
Virtual block: 5d550000 - 5d550000 (size 00000000)
Virtual block: 5d760000 - 5d760000 (size 00000000)
Virtual block: 5e170000 - 5e170000 (size 00000000)
Virtual block: 5eb80000 - 5eb80000 (size 00000000)
Virtual block: 5ed90000 - 5ed90000 (size 00000000)
Virtual block: 5f9a0000 - 5f9a0000 (size 00000000)
Virtual block: 60040000 - 60040000 (size 00000000)
Virtual block: 60250000 - 60250000 (size 00000000)
Virtual block: 60e70000 - 60e70000 (size 00000000)
Virtual block: 621e0000 - 621e0000 (size 00000000)
Virtual block: 623f0000 - 623f0000 (size 00000000)
Virtual block: 62690000 - 62690000 (size 00000000)
Virtual block: 62b30000 - 62b30000 (size 00000000)
Virtual block: 62d40000 - 62d40000 (size 00000000)
Virtual block: 648f0000 - 648f0000 (size 00000000)
Virtual block: 64c90000 - 64c90000 (size 00000000)
Virtual block: 64ea0000 - 64ea0000 (size 00000000)
Virtual block: 66ad0000 - 66ad0000 (size 00000000)
Virtual block: 676a0000 - 676a0000 (size 00000000)
Virtual block: 678b0000 - 678b0000 (size 00000000)
Virtual block: 682c0000 - 682c0000 (size 00000000)
Virtual block: 69100000 - 69100000 (size 00000000)
Virtual block: 68760000 - 68760000 (size 00000000)
Virtual block: 6ada0000 - 6ada0000 (size 00000000)
Virtual block: 6afb0000 - 6afb0000 (size 00000000)
Virtual block: 6b1c0000 - 6b1c0000 (size 00000000)
Virtual block: 6b4d0000 - 6b4d0000 (size 00000000)
Virtual block: 6bcb0000 - 6bcb0000 (size 00000000)
Virtual block: 6bec0000 - 6bec0000 (size 00000000)
Virtual block: 6c630000 - 6c630000 (size 00000000)
Virtual block: 6e0c0000 - 6e0c0000 (size 00000000)
Virtual block: 6da00000 - 6da00000 (size 00000000)
Virtual block: 6e8e0000 - 6e8e0000 (size 00000000)
Virtual block: 6dcf0000 - 6dcf0000 (size 00000000)
Virtual block: 6e2d0000 - 6e2d0000 (size 00000000)
Virtual block: 6e5a0000 - 6e5a0000 (size 00000000)
Virtual block: 6e790000 - 6e790000 (size 00000000)
Virtual block: 6ecf0000 - 6ecf0000 (size 00000000)
Virtual block: 6ee00000 - 6ee00000 (size 00000000)
Virtual block: 6f000000 - 6f000000 (size 00000000)
Virtual block: 6f740000 - 6f740000 (size 00000000)
Virtual block: 6f460000 - 6f460000 (size 00000000)
Virtual block: 6fb60000 - 6fb60000 (size 00000000)
Virtual block: 70280000 - 70280000 (size 00000000)
Virtual block: 70490000 - 70490000 (size 00000000)
Virtual block: 70780000 - 70780000 (size 00000000)
Virtual block: 714e0000 - 714e0000 (size 00000000)
Virtual block: 716f0000 - 716f0000 (size 00000000)
Virtual block: 71d80000 - 71d80000 (size 00000000)
Virtual block: 73220000 - 73220000 (size 00000000)
Virtual block: 73430000 - 73430000 (size 00000000)
Virtual block: 74550000 - 74550000 (size 00000000)
Virtual block: 79d50000 - 79d50000 (size 00000000)
Virtual block: 73bb0000 - 73bb0000 (size 00000000)
Virtual block: 7b730000 - 7b730000 (size 00000000)
Virtual block: 7c140000 - 7c140000 (size 00000000)
Virtual block: 74f50000 - 74f50000 (size 00000000)
Virtual block: 7c450000 - 7c450000 (size 00000000)
Virtual block: 7d7c0000 - 7d7c0000 (size 00000000)
Virtual block: 7d9d0000 - 7d9d0000 (size 00000000)
Virtual block: 7ebe0000 - 7ebe0000 (size 00000000)
Virtual block: 7f980000 - 7f980000 (size 00000000)
Virtual block: 7fb90000 - 7fb90000 (size 00000000)
028b0000 00001002 80192 72580 80192 282 327 9 66 5e LFH
035e0000 00001002 256 176 256 1 9 1 0 0 LFH
Virtual block: 09c20000 - 09c20000 (size 00000000)
03710000 00001002 136864 134140 136864 313 691 13 1 2 LFH
02880000 00001002 64 12 64 4 2 1 0 0
028a0000 00001002 64 4 64 2 1 1 0 0
03a90000 00001002 256 4 256 2 1 1 0 0
03680000 00001002 1088 148 1088 4 3 2 0 0 LFH
03c20000 00001002 1088 276 1088 11 3 2 0 0 LFH
03dd0000 00001002 1088 160 1088 4 7 2 0 0 LFH
03dc0000 00001002 1088 148 1088 6 3 2 0 0 LFH
Virtual block: 5a7f0000 - 5a7f0000 (size 00000000)
Virtual block: 5d030000 - 5d030000 (size 00000000)
Virtual block: 5d1c0000 - 5d1c0000 (size 00000000)
Virtual block: 5feb0000 - 5feb0000 (size 00000000)
Virtual block: 62050000 - 62050000 (size 00000000)
Virtual block: 629a0000 - 629a0000 (size 00000000)
Virtual block: 64b00000 - 64b00000 (size 00000000)
Virtual block: 66de0000 - 66de0000 (size 00000000)
Virtual block: 685d0000 - 685d0000 (size 00000000)
Virtual block: 69b10000 - 69b10000 (size 00000000)
Virtual block: 6b880000 - 6b880000 (size 00000000)
Virtual block: 6df30000 - 6df30000 (size 00000000)
Virtual block: 6f2d0000 - 6f2d0000 (size 00000000)
Virtual block: 700f0000 - 700f0000 (size 00000000)
Virtual block: 71350000 - 71350000 (size 00000000)
Virtual block: 71900000 - 71900000 (size 00000000)
Virtual block: 74760000 - 74760000 (size 00000000)
Virtual block: 750d0000 - 750d0000 (size 00000000)
Virtual block: 7d630000 - 7d630000 (size 00000000)
Virtual block: 7f7f0000 - 7f7f0000 (size 00000000)
Virtual block: 05a30000 - 05a30000 (size 00000000)
03ea0000 00001002 1920 1568 1920 10 4 4 21 0 LFH
03e90000 00001002 1088 280 1088 2 2 2 0 0 LFH
04280000 00001002 256 160 256 2 4 1 0 0 LFH
04240000 00001002 256 172 256 126 3 1 0 0
096d0000 00001002 256 148 256 2 4 1 0 0 LFH
041c0000 00001002 1088 244 1088 11 2 2 0 0 LFH
0f5d0000 00001002 3540 3504 3540 607 48 6 0 0 LFH
External fragmentation 17 % (48 free blocks)
0f580000 00011002 256 4 256 1 2 1 0 0
23030000 00001002 64 4 64 1 2 1 0 0
231c0000 00001002 64 8 64 1 1 1 0 0
23380000 00001002 256 4 256 1 2 1 0 0
26ac0000 00001003 256 76 256 49 13 1 0 N/A
26b60000 00001003 256 4 256 2 1 1 0 N/A
26d50000 00001003 256 4 256 2 1 1 0 N/A
26f10000 00001003 256 4 256 2 1 1 0 N/A
26bf0000 00001003 256 4 256 2 1 1 0 N/A
270e0000 00001003 256 4 256 2 1 1 0 N/A
26ed0000 00001003 256 4 256 2 1 1 0 N/A
27090000 00001003 256 44 256 23 20 1 0 N/A
26fd0000 00001003 256 4 256 2 1 1 0 N/A
26f50000 00001003 256 4 256 2 1 1 0 N/A
276f0000 00001003 256 4 256 2 1 1 0 N/A
27050000 00001003 256 4 256 2 1 1 0 N/A
278f0000 00001003 256 4 256 2 1 1 0 N/A
27690000 00001003 256 4 256 2 1 1 0 N/A
58d20000 00001002 64 32 64 21 1 1 0 0
5abc0000 00001002 15424 8792 15424 6 11 5 0 0 LFH
5ab70000 00001002 1088 188 1088 6 3 2 0 0 LFH
5ad60000 00001002 1088 148 1088 9 3 2 0 0 LFH
5b380000 00001002 1088 196 1088 12 4 2 0 0 LFH
5aaf0000 00001002 1088 220 1088 2 4 2 0 0 LFH
5ab90000 00001002 64 24 64 2 2 1 0 0
5d020000 00001002 15424 10788 15424 22 7 5 0 0 LFH
10180000 00001002 1088 616 1088 82 2 2 0 0 LFH
10b70000 00001002 256 4 256 2 1 1 0 0
最大的两个堆块，一个80192K 136864K 才这么小。这个数据没统计堆块里的Virtual block大小。看看加VirtualBlock多大
0:000> !heap -stat -h 028b0000
heap @ 028b0000
group-by: TOTSIZE max-display: 20
size #blocks total ( %) (percent of total busy bytes)
200000 14 - 2800000 (22.96)
200010 13 - 2600130 (21.81)
10000 209 - 2090000 (18.69)
41410 7e - 201dfe0 (18.44)
100000 14 - 1400000 (11.48)
109c10 7 - 744470 (4.17)
fee8 15 - 14e908 (0.75)
b540 15 - ede40 (0.53)
ec 63b - 5be64 (0.21)
10 5126 - 51260 (0.18)
94 8bc - 50cb0 (0.18)
18c 114 - 1aaf0 (0.06)
214 c4 - 19750 (0.06)
1000 14 - 14000 (0.04)
16c bd - 10cbc (0.04)
bc 15b - fed4 (0.04)
b4 13d - dee4 (0.03)
90 10e - 97e0 (0.02)
550 15 - 6f90 (0.02)
4b0 15 - 6270 (0.01)
2800000 / 22.96 * 100 = 182M
0:000> !heap -stat -h 03710000
heap @ 03710000
group-by: TOTSIZE max-display: 20
size #blocks total ( %) (percent of total busy bytes)
50 2b39d - d82110 (11.91)
7d054 19 - c35834 (10.76)
5c 17265 - 851c4c (7.33)
20 3f285 - 7e50a0 (6.96)
30 23e48 - 6bad80 (5.93)
18 4693e - 69ddd0 (5.83)
28 1d651 - 497ca8 (4.05)
488279 1 - 488279 (3.99)
48 baed - 3492a8 (2.90)
14 22640 - 2afd00 (2.37)
1a8 1825 - 27fd48 (2.20)
10 24d8f - 24d8f0 (2.03)
24 c62d - 1bde54 (1.54)
fc 1b9b - 1b2c94 (1.50)
4c90 5a - 1aeaa0 (1.48)
58 4dda - 1ac2f0 (1.47)
10000 1a - 1a0000 (1.43)
68 3a1a - 179a90 (1.30)
8fc0 2a - 179580 (1.30)
3c 5e7b - 1624d4 (1.22)
D82110/11.91*100 = 118M
182+118 也才300M 其他的 1.3G哪去了？
第二步 全地址排查
使用!heap命令统计的都是 使用new 操作管理的内存，使用malloc等其他一些手段弄出来的内存是不受heap管理的。要看全内存 需要用到另外一个命令了
!address
0:000> !address
Mapping file section regions...
Mapping module regions...
Mapping PEB regions...
Mapping TEB and stack regions...
*** WARNING: Unable to verify checksum for h264dec.dll
*** ERROR: Symbol file could not be found. Defaulted to export symbols for h264dec.dll -
Mapping heap regions...
Mapping page heap regions...
Mapping other regions...
Mapping stack trace database regions...
Mapping activation context regions...
BaseAddr EndAddr+1 RgnSize Type State Protect Usage
---------------------------------------------------------------------------------------------
+ 0 10000 10000 <unknown>
+ 10000 20000 10000 Heap [ID: 1; Handle: 00010000; Type: Segment]
+ 20000 21000 1000 <unknown>
+ 21000 30000 f000 <unknown>
+ 30000 111000 e1000 Stack [~0; 6c8.aec]
+ 111000 130000 1f000 Stack [~0; 6c8.aec]
+ 130000 134000 4000 Other [System Default Activation Context Data]
+ 134000 140000 c000 <unknown>
+ 140000 141000 1000 Other [Activation Context Data]
+ 141000 150000 f000 <unknown>
+ 150000 151000 1000 Other [Switch Context Data]
+ 151000 160000 f000 <unknown>
+ 160000 1c7000 67000 <unknown>
+ 1c7000 1d0000 9000 <unknown>
+ 1d0000 1d1000 1000 <unknown>
+ 1d1000 1e0000 f000 <unknown>
+ 1e0000 1e1000 1000 Image [libcurl; "C:\Program Files\DSS_F3.0_Client\libcurl.dll"]
+ 1e1000 211000 30000 Image [libcurl; "C:\Program Files\DSS_F3.0_Client\libcurl.dll"]
+ 211000 21a000 9000 Image [libcurl; "C:\Program Files\DSS_F3.0_Client\libcurl.dll"]
+ 21a000 21b000 1000 Image [libcurl; "C:\Program Files\DSS_F3.0_Client\libcurl.dll"]
+ 21b000 21f000 4000 Image [libcurl; "C:\Program Files\DSS_F3.0_Client\libcurl.dll"]
+ 21f000 220000 1000 <unknown>
+ 220000 221000 1000 <unknown>
+ 221000 230000 f000 <unknown>
+ 230000 231000 1000 Image [DHTalkFtpDll; "C:\Program Files\DSS_F3.0_Client\DHTalkFtpDll.dll"]
+ 231000 240000 f000 Image [DHTalkFtpDll; "C:\Program Files\DSS_F3.0_Client\DHTalkFtpDll.dll"]
+ 240000 245000 5000 Image [DHTalkFtpDll; "C:\Program Files\DSS_F3.0_Client\DHTalkFtpDll.dll"]
+ 245000 246000 1000 Image [DHTalkFtpDll; "C:\Program Files\DSS_F3.0_Client\DHTalkFtpDll.dll"]
+ 246000 249000 3000 Image [DHTalkFtpDll; "C:\Program Files\DSS_F3.0_Client\DHTalkFtpDll.dll"]
+ 249000 250000 7000 <unknown>
+ 250000 251000 1000 <unknown>
+ 251000 260000 f000 <unknown>
+ 260000 261000 1000 Image [TTSSpeaker; "C:\Program Files\DSS_F3.0_Client\TTSSpeaker.dll"]
+ 261000 263000 2000 Image [TTSSpeaker; "C:\Program Files\DSS_F3.0_Client\TTSSpeaker.dll"]
+ 263000 265000 2000 Image [TTSSpeaker; "C:\Program Files\DSS_F3.0_Client\TTSSpeaker.dll"]
+ 265000 266000 1000 Image [TTSSpeaker; "C:\Program Files\DSS_F3.0_Client\TTSSpeaker.dll"]
+ 266000 268000 2000 Image [TTSSpeaker; "C:\Program Files\DSS_F3.0_Client\TTSSpeaker.dll"]
+ 268000 270000 8000 <unknown>
+ 270000 271000 1000 <unknown>
+ 271000 280000 f000 <unknown>
+ 280000 281000 1000 Image [ProgressDlg; "C:\Program Files\DSS_F3.0_Client\ProgressDlg.dll"]
+ 281000 2ac000 2b000 Image [ProgressDlg; "C:\Program Files\DSS_F3.0_Client\ProgressDlg.dll"]
+ 2ac000 2b7000 b000 Image [ProgressDlg; "C:\Program Files\DSS_F3.0_Client\ProgressDlg.dll"]
+ 2b7000 2bb000 4000 Image [ProgressDlg; "C:\Program Files\DSS_F3.0_Client\ProgressDlg.dll"]
+ 2bb000 2bc000 1000 Image [ProgressDlg; "C:\Program Files\DSS_F3.0_Client\ProgressDlg.dll"]
+ 2bc000 2bf000 3000 Image [ProgressDlg; "C:\Program Files\DSS_F3.0_Client\ProgressDlg.dll"]
+ 2bf000 2df000 20000 Image [ProgressDlg; "C:\Program Files\DSS_F3.0_Client\ProgressDlg.dll"]
+ 2df000 2e0000 1000 <unknown>
+ 2e0000 3e0000 100000 Heap [ID: 0; Handle: 002e0000; Type: Segment]
+ 3e0000 3e1000 1000 <unknown>
+ 3e1000 3f0000 f000 <unknown>
+ 3f0000 3f1000 1000 <unknown>
+ 3f1000 400000 f000 <unknown>
这里对第七列 Usage 做一下解释
HEAP 堆管理器托管
STACK 栈内存
Image 放二进制代码的位置
TEB 操作系统管理线程用的内存
<Unknow> 除new方法外向系统获取的内存
HEAP 不用看了!heap 开头的命令可以详细分析 才300~400M
IMAGE 程序启动后是固定的，也不用看。看看STACK.
每个线程都有自己的STACK
在Notepad++里搜索STACK看看
Search "Stack" (394 hits in 1 file)
\\10.30.21.101\UserDesktop01\12366\DeskTop\3S大法查不出来的内存泄露如何查.txt (394 hits)
Line 7: + 30000 111000 e1000 Stack [~0; 6c8.aec]
Line 8: + 111000 130000 1f000 Stack [~0; 6c8.aec]
Line 263: + 4590000 468f000 ff000 Stack [~1; 6c8.19c]
Line 264: + 468f000 4690000 1000 Stack [~1; 6c8.19c]
……
[~0; 6c8.aec] 这个6c8是进程ID aec是线程ID
每个线程占了两个 这样应该由 (394 hits)/2 = 196个线程
一个线程用了1M 大概是 200M内存加上前面堆的400M 有600M左右
还有1G的肯定是在<Unknow> 里了
Notepad++ 搜索<Unknow> 看第三列大小 找最大的
Line 1138: + 39cf0000 48b9c000 eeac000 <unknown>
eeac000 = 238M
然后NotePad++搜索 eeac000
Search "eeac000" (3 hits in 1 file)
\\10.30.21.101\UserDesktop01\12366\DeskTop\3S大法查不出来的内存泄露如何查.txt (3 hits)
Line 1117: + 29c00000 38aac000 eeac000 Line 1138: + 39cf0000 48b9c000 eeac000 Line 1147: + 49c90000 58b3c000 eeac000 <unknown>
有三个 共714M 大头就是他们了
搜下他们被谁用了
0:000> s -d 0 l?-1 29c00000
246ad038 29c00000 246ad0c0 00000000 00000001 ...)..j$........
2500b3dc 29c00000 4ac00000 69c00000 88c00000 ...)...J...i....
2680db50 29c00000 00000cc0 244e13a0 00000001 ...)......N$....
2682045c 29c00000 00000000 2680db40 00000000 ...)[email protected]&....
0:000> s -d 0 l?-1 39cf0000
0f6aabb8 39cf0000 00000dd4 244e13a0 00000001 ...9......N$....
2924d39c 39cf0000 00000000 0f6aaba8 00000000 ...9......j.....
0:000> s -d 0 l?-1 49c90000
0f925288 49c90000 201c5c06 57286b63 653e64ad ...I.\. ck(W.d>e
267ad5d8 49c90000 49c88000 49b90000 00010007 ...I...I...I....
2924d48c 49c90000 00000000 39ccdf88 00000000 ...I.......9....
39ccdf98 49c90000 00000c90 244e13a0 00000001 ...I......N$....
找看起来差不多的看看
0:000> !heap -x 2682045c
Entry User Heap Segment Size PrevSize Unused Flags
-----------------------------------------------------------------------------
26820410 26820418 002e0000 0f6f9ea8 78 - c LFH;busy
0:000> !heap -x 2924d39c
Entry User Heap Segment Size PrevSize Unused Flags
-----------------------------------------------------------------------------
2924d350 2924d358 002e0000 0f6fd130 78 - c LFH;busy
都是0x78大小的，有戏
0:000> dps 26820418
26820418 1cc08380 mshtml!RefCounted<CImgBits,MultiThreadedRefCount>::`vftable'
2682041c 00000000
26820420 00000000
26820424 00000000
26820428 00000000
2682042c 2680db18
26820430 ffffffff
26820434 00000000
26820438 00000000
2682043c 00000000
26820440 00000000
Mshtml 和ie 有关 CImgBits和位图有关。
机智的我告诉了客户端开发人员袁文茂两个关键字 IE 和 图片。
同样机智的立刻想到了电子地图加载了一个9M的jpeg文件。
马上去掉了对此文件的加载，瞬间客户端内存降下去了，世界立刻清净了。
第二天 罪魁祸首的那张图片传归来了，分辨率是 9523*6570
算了一下在内存中的占用 为9523*6570*4 = 238M,果然罪魁祸首就是他，还一次创建了3个。