作者abc990a (yoyoman)
看板Perl
标题[问题] 如何清除杂凑以释放记忆体
时间Wed Oct 16 19:43:40 2024
大家好
先前很感谢gugod大的回覆
这次的问题是如何清除整个杂凑使Perl释放记忆体给OS, 程式流程大略为:
在回圈里面读档进杂凑, 将杂凑内每笔资料分类後输出资料至硬碟, fork些子程序计算数据
程式如下:
foreach my $FH (($FH_1, $FH_2...)) {
my (%Data_1, %Data_2, %Data_3);
读$FH, 分类至各个杂凑...
输出至硬碟...
}
fork一堆子程序...
在结束读档回圈後, 回圈内的三个%Data都还存在, 原先以为fork後父程序的所有变数会完
全复制一份给子程序使记忆体爆掉(但发现其实子程序会透过COW机制先共用父程序的变数?)
,
於是开始尝试将前面的杂凑清空以释放可用的记忆体, 但不管我怎麽试都无法释放先前杂凑
所使用到的记忆体...
在ClearHash_Test.pl的测试中发现简单的值(直接以数字或文字赋值)能透过undef()删除且
释放一部份记忆体, 而复杂的值却无法被直接删除
, 从Dump_Result可看到Perl在赋值使用到纯量变数时会触发COW机制, 但让我不懂的是为何
$Hash{'3'}无法透过undef()删除, 它不是已经跟$Str是不同东西了吗?
OS:Ubuntu 18.04.6, Perl_Ver:5.26.1
ClearHash_Test.pl:
use strict;
use warnings;
use Devel::Peek;
my $Str = 'AAAAAAAAAA' x 10000000;
#Before exe, Mem:1.55G
#After exe, Mem:1.74G, $Str: ~=0.19G
my %Hash;
$Hash{'1'} = 'AAAAAAAAAA' x 10000000;
#Before exe, Mem:1.55G
#After exe, Mem:1.93G, $Hash{'1'} = 'AAAAAAAAAA' x 10000000: ~=0.19G
$Hash{'2'} = $Str;
#Before exe, Mem:1.55G
#After exe, Mem:1.93G, $Hash{'2'} = $Str:0G
$Hash{'3'} = $Str . '123132';
#Before exe, Mem:1.55G
#After exe, Mem:2.12G, $Hash{'3'} = $Str . '123132': ~=0.19G
Dump(%Hash);
Dump($Str);
undef(%Hash);
#Before exe, Mem:1.55G
#After exe, Mem:1.93G, Hash{'3'}的值还留在记忆体
undef($Str);
#Before exe, Mem:1.55G
#After exe, Mem:1.84G
sleep(10000);
Dump_Result(为方便显示$Str已缩短):
%Hash:
SV = PVHV(0x561b3d6b1f70) at 0x561b3d6c9a80
REFCNT = 1
FLAGS = (SHAREKEYS)
ARRAY = 0x561b3d6efa90 (0:5, 1:3)
hash quality = 150.0%
KEYS = 3
FILL = 3
MAX = 7
Elt "1" HASH = 0xe77f895
SV = PV(0x561b3d6aae60) at 0x561b3d6aa1e0
REFCNT = 1
FLAGS = (POK,pPOK)
PV = 0x561b3d7c4be0 "AAAAAAAAAA"\0
CUR = 10
LEN = 12
Elt "2" HASH = 0x10349798
SV = PV(0x561b3d6aae80) at 0x561b3d6aa3c0
REFCNT = 1
FLAGS = (POK,IsCOW,pPOK)
PV = 0x561b3d6ceb40 "AAAAAAAAAA"\0
CUR = 10
LEN = 12
COW_REFCNT = 1
Elt "3" HASH = 0xf299ae6
SV = PV(0x561b3d6aaea0) at 0x561b3d6c90f0
REFCNT = 1
FLAGS = (POK,IsCOW,pPOK)
PV = 0x561b3d6d72f0 "AAAAAAAAAA123132"\0
CUR = 16
LEN = 24
COW_REFCNT = 1
$Str:
SV = PV(0x561b3d6aada0) at 0x561b3d6c9a38
REFCNT = 1
FLAGS = (POK,IsCOW,pPOK)
PV = 0x561b3d6ceb40 "AAAAAAAAAA"\0
CUR = 10
LEN = 12
COW_REFCNT = 1
在这里我想说会不会是COW_REFCNT阻碍GC机制，所以我在$Hash{$i}的赋值上做一点改变，
但读档结束後%Hash依然无法被删除
Read2Hash.pl:
use strict;
use warnings;
use Devel::Peek;
my %Hash;
open(my $FH, '<', $ARGV[0]);
#档案约3.4G
while (my $Line = <$FH>) {
CORE::state $i = 1;
$Hash{$i} = $Line . '';
#此处无IsCOW flag, Mem消耗~=4.28G
#$Hash{$i} = $Line;
#此处有IsCOW flag, Mem消耗~=4.44G
$i++;
undef($Line);
}
close($FH);
print "done.\n";
<STDIN>;
#undef(%Hash);
#Dump(%Hash);
sleep(10000);
网路上有人说能利用子程序来处理不同部份
, 反正子程序正常结束记忆体一定会还给OS
, 但
真的拿它们没办法吗?
--
※ 发信站: 批踢踢实业坊(ptt.cc), 来自: 1.163.195.62 (台湾)
※ 文章网址: https://webptt.com/cn.aspx?n=bbs/Perl/M.1729079022.A.900.html