作者abc990a (yoyoman)
看板Perl
標題[問題] 如何清除雜湊以釋放記憶體
時間Wed Oct 16 19:43:40 2024
大家好
先前很感謝gugod大的回覆
這次的問題是如何清除整個雜湊使Perl釋放記憶體給OS, 程式流程大略為:
在迴圈裡面讀檔進雜湊, 將雜湊內每筆資料分類後輸出資料至硬碟, fork些子程序計算數據
程式如下:
foreach my $FH (($FH_1, $FH_2...)) {
my (%Data_1, %Data_2, %Data_3);
讀$FH, 分類至各個雜湊...
輸出至硬碟...
}
fork一堆子程序...
在結束讀檔迴圈後, 迴圈內的三個%Data都還存在, 原先以為fork後父程序的所有變數會完
全複製一份給子程序使記憶體爆掉(但發現其實子程序會透過COW機制先共用父程序的變數?)
,
於是開始嘗試將前面的雜湊清空以釋放可用的記憶體, 但不管我怎麼試都無法釋放先前雜湊
所使用到的記憶體...
在ClearHash_Test.pl的測試中發現簡單的值(直接以數字或文字賦值)能透過undef()刪除且
釋放一部份記憶體, 而複雜的值卻無法被直接刪除
, 從Dump_Result可看到Perl在賦值使用到純量變數時會觸發COW機制, 但讓我不懂的是為何
$Hash{'3'}無法透過undef()刪除, 它不是已經跟$Str是不同東西了嗎?
OS:Ubuntu 18.04.6, Perl_Ver:5.26.1
ClearHash_Test.pl:
use strict;
use warnings;
use Devel::Peek;
my $Str = 'AAAAAAAAAA' x 10000000;
#Before exe, Mem:1.55G
#After exe, Mem:1.74G, $Str: ~=0.19G
my %Hash;
$Hash{'1'} = 'AAAAAAAAAA' x 10000000;
#Before exe, Mem:1.55G
#After exe, Mem:1.93G, $Hash{'1'} = 'AAAAAAAAAA' x 10000000: ~=0.19G
$Hash{'2'} = $Str;
#Before exe, Mem:1.55G
#After exe, Mem:1.93G, $Hash{'2'} = $Str:0G
$Hash{'3'} = $Str . '123132';
#Before exe, Mem:1.55G
#After exe, Mem:2.12G, $Hash{'3'} = $Str . '123132': ~=0.19G
Dump(%Hash);
Dump($Str);
undef(%Hash);
#Before exe, Mem:1.55G
#After exe, Mem:1.93G, Hash{'3'}的值還留在記憶體
undef($Str);
#Before exe, Mem:1.55G
#After exe, Mem:1.84G
sleep(10000);
Dump_Result(為方便顯示$Str已縮短):
%Hash:
SV = PVHV(0x561b3d6b1f70) at 0x561b3d6c9a80
REFCNT = 1
FLAGS = (SHAREKEYS)
ARRAY = 0x561b3d6efa90 (0:5, 1:3)
hash quality = 150.0%
KEYS = 3
FILL = 3
MAX = 7
Elt "1" HASH = 0xe77f895
SV = PV(0x561b3d6aae60) at 0x561b3d6aa1e0
REFCNT = 1
FLAGS = (POK,pPOK)
PV = 0x561b3d7c4be0 "AAAAAAAAAA"\0
CUR = 10
LEN = 12
Elt "2" HASH = 0x10349798
SV = PV(0x561b3d6aae80) at 0x561b3d6aa3c0
REFCNT = 1
FLAGS = (POK,IsCOW,pPOK)
PV = 0x561b3d6ceb40 "AAAAAAAAAA"\0
CUR = 10
LEN = 12
COW_REFCNT = 1
Elt "3" HASH = 0xf299ae6
SV = PV(0x561b3d6aaea0) at 0x561b3d6c90f0
REFCNT = 1
FLAGS = (POK,IsCOW,pPOK)
PV = 0x561b3d6d72f0 "AAAAAAAAAA123132"\0
CUR = 16
LEN = 24
COW_REFCNT = 1
$Str:
SV = PV(0x561b3d6aada0) at 0x561b3d6c9a38
REFCNT = 1
FLAGS = (POK,IsCOW,pPOK)
PV = 0x561b3d6ceb40 "AAAAAAAAAA"\0
CUR = 10
LEN = 12
COW_REFCNT = 1
在這裡我想說會不會是COW_REFCNT阻礙GC機制，所以我在$Hash{$i}的賦值上做一點改變，
但讀檔結束後%Hash依然無法被刪除
Read2Hash.pl:
use strict;
use warnings;
use Devel::Peek;
my %Hash;
open(my $FH, '<', $ARGV[0]);
#檔案約3.4G
while (my $Line = <$FH>) {
CORE::state $i = 1;
$Hash{$i} = $Line . '';
#此處無IsCOW flag, Mem消耗~=4.28G
#$Hash{$i} = $Line;
#此處有IsCOW flag, Mem消耗~=4.44G
$i++;
undef($Line);
}
close($FH);
print "done.\n";
<STDIN>;
#undef(%Hash);
#Dump(%Hash);
sleep(10000);
網路上有人說能利用子程序來處理不同部份
, 反正子程序正常結束記憶體一定會還給OS
, 但
真的拿它們沒辦法嗎?
--
※ 發信站: 批踢踢實業坊(ptt.cc), 來自: 1.163.195.62 (臺灣)
※ 文章網址: https://webptt.com/m.aspx?n=bbs/Perl/M.1729079022.A.900.html