2023-11-01 08:55:22,707 INFO  [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins
2023-11-01 08:55:22,995 WARN  [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
2023-11-01 08:55:23,093 INFO  [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2023-11-01 08:55:23,151 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-11-01 08:55:23,158 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-11-01 08:55:23,174 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=125, ProcessCount=170, AvailableMemoryMB=11517
2023-11-01 08:55:23,176 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-11-01 08:55:23,176 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-11-01 08:55:23,305 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1018916932:84 , srv105710637:84 , srv1067594788:84 , srv1068973446:84 , srv1071119844:84 , srv1082807721:84 , srv1107294494:84 , srv1119737969:84 , srv1188472523:84 , srv1205105054:84 , srv120521674:84 , srv1245137732:84 , srv1246480938:84 , srv1274568401:84 , srv1277157739:84 , srv1295098778:84 , srv1300769512:84 , srv1321121208:84 , srv1353088442:84 , srv1386779311:84 , srv141454514:84 , srv142181724:84 , srv1460588460:84 , srv1508337132:84 , srv1531027481:84 , srv1531710177:84 , srv1533253444:84 , srv1612564290:84 , srv161851012:84 , srv1623941822:84 , srv1640410386:84 , srv1642196904:84 , srv1643186708:84 , srv1657212870:84 , srv1658267909:84 , srv166041804:84 , srv1666924533:84 , srv1676154559:84 , srv1683949484:84 , srv1701261773:84 , srv1720981826:84 , srv1737338140:84 , srv1741435981:84 , srv1759036867:84 , srv1763363853:84 , srv1769062096:84 , srv1827662603:84 , srv1853812226:84 , srv1863265544:84 , srv1868202707:84 , srv1916313485:84 , srv1917179960:84 , srv1935519225:84 , srv1940251433:84 , srv1945627428:84 , srv1960891948:84 , srv1972172991:84 , srv1987460246:84 , srv1988586712:84 , srv1990765450:84 , srv2004497819:84 , srv2029693782:84 , srv2045020145:84 , srv2047285106:84 , srv2064842305:84 , srv2084281669:84 , srv220505498:84 , srv234253178:84 , srv27978573:84 , srv289451626:84 , srv297472990:84 , srv339937824:84 , srv368808277:84 , srv38522392:84 , srv412692555:84 , srv420142783:84 , srv421861027:84 , srv42961684:84 , srv467517431:84 , srv479408238:84 , srv51332168:84 , srv522895109:84 , srv530119375:84 , srv535400107:84 , srv578800392:84 , srv590495284:84 , srv598351471:84 , srv612805008:84 , srv615451105:84 , srv622071650:84 , srv626150897:84 , srv628121863:84 , srv702862856:84 , srv755011886:84 , srv922699842:84 , srv939444168:84 , srv956387803:84 , srv987516475:84 , srv990680792:84 , srv1203623539:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90]
2023-11-01 08:55:23,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-11-01 08:55:23,318 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv702862856=93, srv297472990=71, srv939444168=96, srv1246480938=13, srv628121863=92, srv612805008=88, srv615451105=89, srv141454514=21, srv1188472523=8, srv1741435981=43, srv1769062096=46, srv578800392=85, srv161851012=29, srv1119737969=7, srv987516475=98, srv1082807721=5, srv467517431=79, srv1658267909=35, srv2029693782=62, srv1737338140=42, srv1683949484=39, srv412692555=75, srv105710637=1, srv990680792=99, srv1107294494=6, srv1853812226=48, srv530119375=83, srv2084281669=66, srv1759036867=44, srv1067594788=2, srv1640410386=31, srv1666924533=37, srv1508337132=24, srv479408238=80, srv1353088442=19, srv420142783=76, srv1916313485=51, srv1321121208=18, srv1676154559=38, srv42961684=78, srv1018916932=0, srv1972172991=57, srv1612564290=28, srv622071650=90, srv1935519225=53, srv1940251433=54, srv755011886=94, srv1071119844=4, srv1643186708=33, srv1386779311=20, srv1300769512=17, srv339937824=72, srv922699842=95, srv1863265544=49, srv1987460246=58, srv1531710177=26, srv1763363853=45, srv289451626=70, srv1531027481=25, srv590495284=86, srv1945627428=55, srv51332168=81, srv956387803=97, srv38522392=74, srv421861027=77, srv598351471=87, srv120521674=11, srv142181724=22, srv1701261773=40, srv1533253444=27, srv626150897=91, srv1868202707=50, srv1205105054=10, srv1245137732=12, srv220505498=67, srv2004497819=61, srv1460588460=23, srv2045020145=63, srv1657212870=34, srv166041804=36, srv1988586712=59, srv1068973446=3, srv1295098778=16, srv1720981826=41, srv27978573=69, srv2047285106=64, srv2064842305=65, srv522895109=82, srv535400107=84, srv1274568401=14, srv1277157739=15, srv1642196904=32, srv1960891948=56, srv1990765450=60, srv1623941822=30, srv234253178=68, srv1917179960=52, srv1827662603=47, srv368808277=73, srv1203623539=9} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-11-01 08:55:23,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-11-01 08:55:23,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-11-01 08:55:23,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-11-01 08:55:23,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-11-01 08:55:23,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-11-01 08:55:23,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-11-01 08:55:23,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-11-01 08:55:23,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-11-01 08:55:23,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-11-01 08:55:23,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-11-01 08:55:23,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-11-01 08:55:23,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-11-01 08:55:23,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-11-01 08:55:23,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-11-01 08:55:23,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-11-01 08:55:23,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-11-01 08:55:23,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-11-01 08:55:23,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-11-01 08:55:23,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-11-01 08:55:23,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-11-01 08:55:23,393 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-11-01 08:55:23,393 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-11-01 08:55:23,393 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-11-01 08:55:23,393 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-11-01 08:55:23,393 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-11-01 08:55:23,393 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-11-01 08:55:23,393 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-11-01 08:55:23,393 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-11-01 08:55:23,393 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-11-01 08:55:23,393 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-11-01 08:55:23,393 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-11-01 08:55:23,394 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-11-01 08:55:23,394 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-11-01 08:55:23,394 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-11-01 08:55:23,394 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-11-01 08:55:23,394 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-11-01 08:55:23,394 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-11-01 08:55:23,394 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-11-01 08:55:23,394 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-11-01 08:55:23,394 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-11-01 08:55:23,394 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-11-01 08:55:23,394 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-11-01 08:55:23,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-11-01 08:55:23,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-11-01 08:55:23,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-11-01 08:55:23,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-11-01 08:55:23,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-11-01 08:55:23,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-11-01 08:55:23,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-11-01 08:55:23,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-11-01 08:55:23,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-11-01 08:55:23,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-11-01 08:55:23,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-11-01 08:55:23,396 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-11-01 08:55:23,396 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-11-01 08:55:23,396 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-11-01 08:55:23,396 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-11-01 08:55:23,396 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-11-01 08:55:23,396 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-11-01 08:55:23,396 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-11-01 08:55:23,396 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-11-01 08:55:23,396 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-11-01 08:55:23,396 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-11-01 08:55:23,396 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-11-01 08:55:23,396 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-11-01 08:55:23,397 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-11-01 08:55:23,397 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-11-01 08:55:23,397 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-11-01 08:55:23,397 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-11-01 08:55:23,397 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-11-01 08:55:23,397 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-11-01 08:55:23,397 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-11-01 08:55:23,397 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-11-01 08:55:23,397 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-11-01 08:55:23,397 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-11-01 08:55:23,397 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-11-01 08:55:23,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-11-01 08:55:23,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-11-01 08:55:23,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-11-01 08:55:23,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-11-01 08:55:23,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-11-01 08:55:23,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-11-01 08:55:23,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-11-01 08:55:23,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-11-01 08:55:23,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-11-01 08:55:23,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-11-01 08:55:23,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-11-01 08:55:23,398 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-11-01 08:55:23,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-11-01 08:55:23,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-11-01 08:55:23,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-11-01 08:55:23,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-11-01 08:55:23,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-11-01 08:55:23,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-11-01 08:55:23,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-11-01 08:55:23,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-11-01 08:55:23,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-11-01 08:55:23,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-11-01 08:55:23,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-11-01 08:55:23,399 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-11-01 08:55:23,400 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-11-01 08:55:23,556 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-01 08:55:23,556 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-11-01 08:55:23,557 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-11-01 08:55:23,942 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 521 ms to try 5000 different iterations.  Found a solution that moves 4663 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.18232183406807967. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04263809602716856, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.055648407255541134, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5181111111111111, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04263809602716856, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17758333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24391666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-01 08:55:23,952 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1827662603:80 , srv141454514:81 , srv166041804:82 , srv522895109:82 , srv535400107:82 , srv939444168:82 , srv105710637:83 , srv1068973446:83 , srv1188472523:83 , srv1642196904:83 , srv1676154559:83 , srv1853812226:83 , srv1917179960:83 , srv2045020145:83 , srv590495284:83 , srv1067594788:84 , srv120521674:84 , srv1245137732:84 , srv1508337132:84 , srv1533253444:84 , srv1612564290:84 , srv1640410386:84 , srv1683949484:84 , srv1916313485:84 , srv1988586712:84 , srv220505498:84 , srv339937824:84 , srv368808277:84 , srv1205105054:85 , srv1460588460:85 , srv1643186708:85 , srv1657212870:85 , srv1701261773:85 , srv1868202707:85 , srv1945627428:85 , srv1987460246:85 , srv2084281669:85 , srv38522392:85 , srv530119375:85 , srv626150897:85 , srv987516475:85 , srv1082807721:86 , srv1107294494:86 , srv1119737969:86 , srv1274568401:86 , srv1295098778:86 , srv1720981826:86 , srv1737338140:86 , srv1741435981:86 , srv1759036867:86 , srv1990765450:86 , srv2029693782:86 , srv234253178:86 , srv42961684:86 , srv622071650:86 , srv702862856:86 , srv990680792:86 , srv1071119844:87 , srv1321121208:87 , srv142181724:87 , srv1531027481:87 , srv1666924533:87 , srv1763363853:87 , srv1935519225:87 , srv1940251433:87 , srv1960891948:87 , srv2047285106:87 , srv467517431:87 , srv1018916932:88 , srv1277157739:88 , srv1386779311:88 , srv1531710177:88 , srv161851012:88 , srv1623941822:88 , srv1972172991:88 , srv2004497819:88 , srv2064842305:88 , srv27978573:88 , srv297472990:88 , srv420142783:88 , srv51332168:88 , srv615451105:88 , srv922699842:88 , srv1246480938:89 , srv1863265544:89 , srv289451626:89 , srv421861027:89 , srv598351471:89 , srv612805008:89 , srv628121863:89 , srv755011886:89 , srv1300769512:90 , srv1769062096:90 , srv412692555:90 , srv479408238:90 , srv1353088442:91 , srv1658267909:93 , srv956387803:94 , srv578800392:95 , srv1203623539:469 }
2023-11-01 08:55:23,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-11-01 08:55:23,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv702862856=93, srv297472990=71, srv939444168=96, srv1246480938=13, srv628121863=92, srv612805008=88, srv615451105=89, srv141454514=21, srv1188472523=8, srv1741435981=43, srv1769062096=46, srv578800392=85, srv161851012=29, srv1119737969=7, srv987516475=98, srv1082807721=5, srv467517431=79, srv1658267909=35, srv2029693782=62, srv1737338140=42, srv1683949484=39, srv412692555=75, srv105710637=1, srv990680792=99, srv1107294494=6, srv1853812226=48, srv530119375=83, srv2084281669=66, srv1759036867=44, srv1067594788=2, srv1640410386=31, srv1666924533=37, srv1508337132=24, srv479408238=80, srv1353088442=19, srv420142783=76, srv1916313485=51, srv1321121208=18, srv1676154559=38, srv42961684=78, srv1018916932=0, srv1972172991=57, srv1612564290=28, srv622071650=90, srv1935519225=53, srv1940251433=54, srv755011886=94, srv1071119844=4, srv1643186708=33, srv1386779311=20, srv1300769512=17, srv339937824=72, srv922699842=95, srv1863265544=49, srv1987460246=58, srv1531710177=26, srv1763363853=45, srv289451626=70, srv1531027481=25, srv590495284=86, srv1945627428=55, srv51332168=81, srv956387803=97, srv38522392=74, srv421861027=77, srv598351471=87, srv120521674=11, srv142181724=22, srv1701261773=40, srv1533253444=27, srv626150897=91, srv1868202707=50, srv1205105054=10, srv1245137732=12, srv220505498=67, srv2004497819=61, srv1460588460=23, srv2045020145=63, srv1657212870=34, srv166041804=36, srv1988586712=59, srv1068973446=3, srv1295098778=16, srv1720981826=41, srv27978573=69, srv2047285106=64, srv2064842305=65, srv522895109=82, srv535400107=84, srv1274568401=14, srv1277157739=15, srv1642196904=32, srv1960891948=56, srv1990765450=60, srv1623941822=30, srv234253178=68, srv1917179960=52, srv1827662603=47, srv368808277=73, srv1203623539=9} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-11-01 08:55:23,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-11-01 08:55:23,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-11-01 08:55:23,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-11-01 08:55:23,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-11-01 08:55:23,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-11-01 08:55:23,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-11-01 08:55:23,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-11-01 08:55:23,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-11-01 08:55:23,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-11-01 08:55:23,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-11-01 08:55:23,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-11-01 08:55:23,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-11-01 08:55:23,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-11-01 08:55:23,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-11-01 08:55:23,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-11-01 08:55:23,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-11-01 08:55:23,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-11-01 08:55:23,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-11-01 08:55:23,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-11-01 08:55:23,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-11-01 08:55:23,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-11-01 08:55:23,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-11-01 08:55:23,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-11-01 08:55:23,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-11-01 08:55:23,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-11-01 08:55:23,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-11-01 08:55:23,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-11-01 08:55:23,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-11-01 08:55:23,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-11-01 08:55:23,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-11-01 08:55:23,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-11-01 08:55:23,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-11-01 08:55:23,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-11-01 08:55:23,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-11-01 08:55:23,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-11-01 08:55:23,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-11-01 08:55:23,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-11-01 08:55:23,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-11-01 08:55:23,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-11-01 08:55:23,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-11-01 08:55:23,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-11-01 08:55:23,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-11-01 08:55:23,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-11-01 08:55:23,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-11-01 08:55:23,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-11-01 08:55:23,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-11-01 08:55:23,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-11-01 08:55:23,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-11-01 08:55:23,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-11-01 08:55:23,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-11-01 08:55:23,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-11-01 08:55:23,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-11-01 08:55:23,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-11-01 08:55:23,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-11-01 08:55:23,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-11-01 08:55:23,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-11-01 08:55:23,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-11-01 08:55:23,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-11-01 08:55:23,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-11-01 08:55:23,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-11-01 08:55:23,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-11-01 08:55:23,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-11-01 08:55:23,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-11-01 08:55:23,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-11-01 08:55:23,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-11-01 08:55:23,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-11-01 08:55:23,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-11-01 08:55:23,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-11-01 08:55:23,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-11-01 08:55:23,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-11-01 08:55:23,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-11-01 08:55:23,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-11-01 08:55:23,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-11-01 08:55:23,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-11-01 08:55:23,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-11-01 08:55:23,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-11-01 08:55:23,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-11-01 08:55:23,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-11-01 08:55:23,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-11-01 08:55:23,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-11-01 08:55:23,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-11-01 08:55:23,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-11-01 08:55:23,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-11-01 08:55:23,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-11-01 08:55:23,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-11-01 08:55:23,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-11-01 08:55:23,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-11-01 08:55:23,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-11-01 08:55:23,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-11-01 08:55:23,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-11-01 08:55:23,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-11-01 08:55:23,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-11-01 08:55:23,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-11-01 08:55:23,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-11-01 08:55:23,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-11-01 08:55:23,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-11-01 08:55:23,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-11-01 08:55:23,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-11-01 08:55:23,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-11-01 08:55:23,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-11-01 08:55:23,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-11-01 08:55:24,099 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04263809602716856, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.055648407255541134, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04263809602716856, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17758333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24391666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-01 08:55:24,099 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-11-01 08:55:24,099 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.18228918304421537, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04263809602716856, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.055648407255541134, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04263809602716856, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17758333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24391666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-11-01 08:55:24,370 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 374 ms to try 5000 different iterations.  Found a solution that moves 2625 regions; Going from a computed imbalance of 0.18228918304421537 to a new imbalance of 0.014641806780598121. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.022492187334354038); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03625263642256906, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.2916666666666667, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.022492187334354038); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011666666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04275, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-01 08:55:24,381 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv578800392:77 , srv956387803:77 , srv1658267909:78 , srv1246480938:81 , srv1300769512:81 , srv1353088442:81 , srv412692555:81 , srv2004497819:82 , srv289451626:82 , srv297472990:82 , srv421861027:82 , srv1018916932:83 , srv1071119844:83 , srv1386779311:83 , srv1531027481:83 , srv1623941822:83 , srv612805008:83 , srv628121863:83 , srv922699842:83 , srv1295098778:84 , srv1666924533:84 , srv1769062096:84 , srv1863265544:84 , srv1935519225:84 , srv1990765450:84 , srv2047285106:84 , srv234253178:84 , srv27978573:84 , srv420142783:84 , srv479408238:84 , srv598351471:84 , srv615451105:84 , srv755011886:84 , srv990680792:84 , srv1119737969:85 , srv1531710177:85 , srv161851012:85 , srv1643186708:85 , srv1759036867:85 , srv1940251433:85 , srv1960891948:85 , srv1972172991:85 , srv467517431:85 , srv51332168:85 , srv105710637:86 , srv1082807721:86 , srv1277157739:86 , srv142181724:86 , srv1460588460:86 , srv1763363853:86 , srv1987460246:86 , srv339937824:86 , srv1107294494:87 , srv1205105054:87 , srv1245137732:87 , srv1321121208:87 , srv1612564290:87 , srv1657212870:87 , srv1720981826:87 , srv1737338140:87 , srv1741435981:87 , srv1868202707:87 , srv1916313485:87 , srv1945627428:87 , srv2029693782:87 , srv2064842305:87 , srv2084281669:87 , srv38522392:87 , srv987516475:87 , srv1068973446:88 , srv1188472523:88 , srv120521674:88 , srv1274568401:88 , srv1533253444:88 , srv1676154559:88 , srv1683949484:88 , srv220505498:88 , srv42961684:88 , srv530119375:88 , srv622071650:88 , srv702862856:88 , srv1067594788:89 , srv1640410386:89 , srv1642196904:89 , srv166041804:89 , srv1701261773:89 , srv1917179960:89 , srv1988586712:89 , srv2045020145:89 , srv368808277:89 , srv535400107:89 , srv626150897:89 , srv1508337132:90 , srv1827662603:90 , srv590495284:90 , srv939444168:90 , srv1853812226:91 , srv522895109:91 , srv141454514:93 , srv1203623539:505 }
2023-11-01 08:55:24,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-11-01 08:55:24,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv702862856=93, srv297472990=71, srv939444168=96, srv1246480938=13, srv628121863=92, srv612805008=88, srv615451105=89, srv141454514=21, srv1188472523=8, srv1741435981=43, srv1769062096=46, srv578800392=85, srv161851012=29, srv1119737969=7, srv987516475=98, srv1082807721=5, srv467517431=79, srv1658267909=35, srv2029693782=62, srv1737338140=42, srv1683949484=39, srv412692555=75, srv105710637=1, srv990680792=99, srv1107294494=6, srv1853812226=48, srv530119375=83, srv2084281669=66, srv1759036867=44, srv1067594788=2, srv1640410386=31, srv1666924533=37, srv1508337132=24, srv479408238=80, srv1353088442=19, srv420142783=76, srv1916313485=51, srv1321121208=18, srv1676154559=38, srv42961684=78, srv1018916932=0, srv1972172991=57, srv1612564290=28, srv622071650=90, srv1935519225=53, srv1940251433=54, srv755011886=94, srv1071119844=4, srv1643186708=33, srv1386779311=20, srv1300769512=17, srv339937824=72, srv922699842=95, srv1863265544=49, srv1987460246=58, srv1531710177=26, srv1763363853=45, srv289451626=70, srv1531027481=25, srv590495284=86, srv1945627428=55, srv51332168=81, srv956387803=97, srv38522392=74, srv421861027=77, srv598351471=87, srv120521674=11, srv142181724=22, srv1701261773=40, srv1533253444=27, srv626150897=91, srv1868202707=50, srv1205105054=10, srv1245137732=12, srv220505498=67, srv2004497819=61, srv1460588460=23, srv2045020145=63, srv1657212870=34, srv166041804=36, srv1988586712=59, srv1068973446=3, srv1295098778=16, srv1720981826=41, srv27978573=69, srv2047285106=64, srv2064842305=65, srv522895109=82, srv535400107=84, srv1274568401=14, srv1277157739=15, srv1642196904=32, srv1960891948=56, srv1990765450=60, srv1623941822=30, srv234253178=68, srv1917179960=52, srv1827662603=47, srv368808277=73, srv1203623539=9} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-11-01 08:55:24,404 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-11-01 08:55:24,404 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-11-01 08:55:24,404 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-11-01 08:55:24,404 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-11-01 08:55:24,404 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-11-01 08:55:24,404 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-11-01 08:55:24,404 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-11-01 08:55:24,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-11-01 08:55:24,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-11-01 08:55:24,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-11-01 08:55:24,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-11-01 08:55:24,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-11-01 08:55:24,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-11-01 08:55:24,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-11-01 08:55:24,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-11-01 08:55:24,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-11-01 08:55:24,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-11-01 08:55:24,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-11-01 08:55:24,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-11-01 08:55:24,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-11-01 08:55:24,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-11-01 08:55:24,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-11-01 08:55:24,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-11-01 08:55:24,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-11-01 08:55:24,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-11-01 08:55:24,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-11-01 08:55:24,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-11-01 08:55:24,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-11-01 08:55:24,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-11-01 08:55:24,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-11-01 08:55:24,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-11-01 08:55:24,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-11-01 08:55:24,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-11-01 08:55:24,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-11-01 08:55:24,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-11-01 08:55:24,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-11-01 08:55:24,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-11-01 08:55:24,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-11-01 08:55:24,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-11-01 08:55:24,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-11-01 08:55:24,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-11-01 08:55:24,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-11-01 08:55:24,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-11-01 08:55:24,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-11-01 08:55:24,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-11-01 08:55:24,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-11-01 08:55:24,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-11-01 08:55:24,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-11-01 08:55:24,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-11-01 08:55:24,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-11-01 08:55:24,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-11-01 08:55:24,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-11-01 08:55:24,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-11-01 08:55:24,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-11-01 08:55:24,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-11-01 08:55:24,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-11-01 08:55:24,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-11-01 08:55:24,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-11-01 08:55:24,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-11-01 08:55:24,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-11-01 08:55:24,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-11-01 08:55:24,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-11-01 08:55:24,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-11-01 08:55:24,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-11-01 08:55:24,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-11-01 08:55:24,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-11-01 08:55:24,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-11-01 08:55:24,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-11-01 08:55:24,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-11-01 08:55:24,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-11-01 08:55:24,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-11-01 08:55:24,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-11-01 08:55:24,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-11-01 08:55:24,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-11-01 08:55:24,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-11-01 08:55:24,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-11-01 08:55:24,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-11-01 08:55:24,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-11-01 08:55:24,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-11-01 08:55:24,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-11-01 08:55:24,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-11-01 08:55:24,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-11-01 08:55:24,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-11-01 08:55:24,410 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-11-01 08:55:24,410 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-11-01 08:55:24,410 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-11-01 08:55:24,410 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-11-01 08:55:24,410 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-11-01 08:55:24,410 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-11-01 08:55:24,410 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-11-01 08:55:24,410 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-11-01 08:55:24,410 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-11-01 08:55:24,410 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-11-01 08:55:24,410 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-11-01 08:55:24,410 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-11-01 08:55:24,410 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-11-01 08:55:24,410 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-11-01 08:55:24,410 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-11-01 08:55:24,410 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-11-01 08:55:24,410 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-11-01 08:55:24,411 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-11-01 08:55:24,531 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.022492187334354038); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03625263642256906, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.022492187334354038); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011666666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04275, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-01 08:55:24,531 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-11-01 08:55:24,531 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.014623426137740763, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.022492187334354038); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03625263642256906, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.022492187334354038); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011666666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04275, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-11-01 08:55:24,798 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 372 ms to try 5000 different iterations.  Found a solution that moves 973 regions; Going from a computed imbalance of 0.014623426137740763 to a new imbalance of 8.40876017214505E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007564001353343065); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008863600391305173); MoveCostFunction : (multiplier=7.0, imbalance=0.10811111111111112, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.007564001353343065); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=4.166666666666667E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00425); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-01 08:55:24,800 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv2064842305:82 , srv1274568401:83 , srv1321121208:83 , srv141454514:83 , srv1701261773:83 , srv1769062096:83 , srv1863265544:83 , srv2029693782:83 , srv51332168:83 , srv702862856:83 , srv1067594788:84 , srv1107294494:84 , srv1277157739:84 , srv142181724:84 , srv1508337132:84 , srv1642196904:84 , srv1720981826:84 , srv1853812226:84 , srv1972172991:84 , srv368808277:84 , srv42961684:84 , srv479408238:84 , srv522895109:84 , srv530119375:84 , srv622071650:84 , srv626150897:84 , srv1205105054:85 , srv120521674:85 , srv1300769512:85 , srv1353088442:85 , srv1531710177:85 , srv161851012:85 , srv1640410386:85 , srv1657212870:85 , srv1676154559:85 , srv1683949484:85 , srv1737338140:85 , srv1741435981:85 , srv1763363853:85 , srv1868202707:85 , srv1945627428:85 , srv1987460246:85 , srv1988586712:85 , srv2045020145:85 , srv2084281669:85 , srv220505498:85 , srv467517431:85 , srv578800392:85 , srv590495284:85 , srv598351471:85 , srv612805008:85 , srv615451105:85 , srv628121863:85 , srv755011886:85 , srv1082807721:86 , srv1188472523:86 , srv1295098778:86 , srv1386779311:86 , srv1533253444:86 , srv1623941822:86 , srv1643186708:86 , srv1658267909:86 , srv1759036867:86 , srv1827662603:86 , srv1916313485:86 , srv1917179960:86 , srv1935519225:86 , srv1940251433:86 , srv1960891948:86 , srv2047285106:86 , srv27978573:86 , srv289451626:86 , srv339937824:86 , srv412692555:86 , srv420142783:86 , srv421861027:86 , srv535400107:86 , srv922699842:86 , srv939444168:86 , srv956387803:86 , srv1018916932:87 , srv105710637:87 , srv1068973446:87 , srv1071119844:87 , srv1119737969:87 , srv1245137732:87 , srv1531027481:87 , srv1612564290:87 , srv166041804:87 , srv1666924533:87 , srv234253178:87 , srv38522392:87 , srv987516475:87 , srv1460588460:88 , srv1990765450:88 , srv2004497819:88 , srv297472990:88 , srv990680792:88 , srv1246480938:89 , srv1203623539:551 }
2023-11-01 08:55:24,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-11-01 08:55:24,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv702862856=93, srv297472990=71, srv939444168=96, srv1246480938=13, srv628121863=92, srv612805008=88, srv615451105=89, srv141454514=21, srv1188472523=8, srv1741435981=43, srv1769062096=46, srv578800392=85, srv161851012=29, srv1119737969=7, srv987516475=98, srv1082807721=5, srv467517431=79, srv1658267909=35, srv2029693782=62, srv1737338140=42, srv1683949484=39, srv412692555=75, srv105710637=1, srv990680792=99, srv1107294494=6, srv1853812226=48, srv530119375=83, srv2084281669=66, srv1759036867=44, srv1067594788=2, srv1640410386=31, srv1666924533=37, srv1508337132=24, srv479408238=80, srv1353088442=19, srv420142783=76, srv1916313485=51, srv1321121208=18, srv1676154559=38, srv42961684=78, srv1018916932=0, srv1972172991=57, srv1612564290=28, srv622071650=90, srv1935519225=53, srv1940251433=54, srv755011886=94, srv1071119844=4, srv1643186708=33, srv1386779311=20, srv1300769512=17, srv339937824=72, srv922699842=95, srv1863265544=49, srv1987460246=58, srv1531710177=26, srv1763363853=45, srv289451626=70, srv1531027481=25, srv590495284=86, srv1945627428=55, srv51332168=81, srv956387803=97, srv38522392=74, srv421861027=77, srv598351471=87, srv120521674=11, srv142181724=22, srv1701261773=40, srv1533253444=27, srv626150897=91, srv1868202707=50, srv1205105054=10, srv1245137732=12, srv220505498=67, srv2004497819=61, srv1460588460=23, srv2045020145=63, srv1657212870=34, srv166041804=36, srv1988586712=59, srv1068973446=3, srv1295098778=16, srv1720981826=41, srv27978573=69, srv2047285106=64, srv2064842305=65, srv522895109=82, srv535400107=84, srv1274568401=14, srv1277157739=15, srv1642196904=32, srv1960891948=56, srv1990765450=60, srv1623941822=30, srv234253178=68, srv1917179960=52, srv1827662603=47, srv368808277=73, srv1203623539=9} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-11-01 08:55:24,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-11-01 08:55:24,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-11-01 08:55:24,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-11-01 08:55:24,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-11-01 08:55:24,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-11-01 08:55:24,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-11-01 08:55:24,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-11-01 08:55:24,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-11-01 08:55:24,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-11-01 08:55:24,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-11-01 08:55:24,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-11-01 08:55:24,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-11-01 08:55:24,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-11-01 08:55:24,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-11-01 08:55:24,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-11-01 08:55:24,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-11-01 08:55:24,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-11-01 08:55:24,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-11-01 08:55:24,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-11-01 08:55:24,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-11-01 08:55:24,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-11-01 08:55:24,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-11-01 08:55:24,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-11-01 08:55:24,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-11-01 08:55:24,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-11-01 08:55:24,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-11-01 08:55:24,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-11-01 08:55:24,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-11-01 08:55:24,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-11-01 08:55:24,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-11-01 08:55:24,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-11-01 08:55:24,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-11-01 08:55:24,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-11-01 08:55:24,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-11-01 08:55:24,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-11-01 08:55:24,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-11-01 08:55:24,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-11-01 08:55:24,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-11-01 08:55:24,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-11-01 08:55:24,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-11-01 08:55:24,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-11-01 08:55:24,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-11-01 08:55:24,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-11-01 08:55:24,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-11-01 08:55:24,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-11-01 08:55:24,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-11-01 08:55:24,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-11-01 08:55:24,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-11-01 08:55:24,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-11-01 08:55:24,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-11-01 08:55:24,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-11-01 08:55:24,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-11-01 08:55:24,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-11-01 08:55:24,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-11-01 08:55:24,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-11-01 08:55:24,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-11-01 08:55:24,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-11-01 08:55:24,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-11-01 08:55:24,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-11-01 08:55:24,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-11-01 08:55:24,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-11-01 08:55:24,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-11-01 08:55:24,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-11-01 08:55:24,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-11-01 08:55:24,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-11-01 08:55:24,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-11-01 08:55:24,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-11-01 08:55:24,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-11-01 08:55:24,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-11-01 08:55:24,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-11-01 08:55:24,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-11-01 08:55:24,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-11-01 08:55:24,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-11-01 08:55:24,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-11-01 08:55:24,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-11-01 08:55:24,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-11-01 08:55:24,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-11-01 08:55:24,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-11-01 08:55:24,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-11-01 08:55:24,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-11-01 08:55:24,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-11-01 08:55:24,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-11-01 08:55:24,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-11-01 08:55:24,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-11-01 08:55:24,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-11-01 08:55:24,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-11-01 08:55:24,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-11-01 08:55:24,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-11-01 08:55:24,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-11-01 08:55:24,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-11-01 08:55:24,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-11-01 08:55:24,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-11-01 08:55:24,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-11-01 08:55:24,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-11-01 08:55:24,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-11-01 08:55:24,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-11-01 08:55:24,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-11-01 08:55:24,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-11-01 08:55:24,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-11-01 08:55:24,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-11-01 08:55:24,822 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-11-01 08:55:24,928 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007564001353343065); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008863600391305173); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.007564001353343065); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=4.166666666666667E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00425); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-01 08:55:24,929 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-11-01 08:55:24,929 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=8.34062925595378E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007564001353343065); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008863600391305173); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.007564001353343065); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=4.166666666666667E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00425); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-11-01 08:55:25,283 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 453 ms to try 5000 different iterations.  Found a solution that moves 251 regions; Going from a computed imbalance of 8.34062925595378E-4 to a new imbalance of 1.967104817489211E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.4668347344853864E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018349396085439343); MoveCostFunction : (multiplier=7.0, imbalance=0.02788888888888889, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.4668347344853864E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-01 08:55:25,284 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1082807721:83 , srv1246480938:83 , srv1460588460:83 , srv38522392:83 , srv987516475:83 , srv1018916932:84 , srv1119737969:84 , srv1188472523:84 , srv1245137732:84 , srv141454514:84 , srv1508337132:84 , srv1531710177:84 , srv1533253444:84 , srv1612564290:84 , srv161851012:84 , srv1640410386:84 , srv166041804:84 , srv1666924533:84 , srv1737338140:84 , srv1741435981:84 , srv1763363853:84 , srv1853812226:84 , srv1917179960:84 , srv1935519225:84 , srv1940251433:84 , srv1960891948:84 , srv1990765450:84 , srv2004497819:84 , srv27978573:84 , srv297472990:84 , srv368808277:84 , srv420142783:84 , srv42961684:84 , srv479408238:84 , srv535400107:84 , srv590495284:84 , srv598351471:84 , srv615451105:84 , srv622071650:84 , srv626150897:84 , srv755011886:84 , srv939444168:84 , srv990680792:84 , srv1067594788:85 , srv1068973446:85 , srv1071119844:85 , srv1107294494:85 , srv1205105054:85 , srv120521674:85 , srv1274568401:85 , srv1277157739:85 , srv1353088442:85 , srv1386779311:85 , srv142181724:85 , srv1531027481:85 , srv1623941822:85 , srv1643186708:85 , srv1657212870:85 , srv1683949484:85 , srv1701261773:85 , srv1720981826:85 , srv1759036867:85 , srv1769062096:85 , srv1868202707:85 , srv1916313485:85 , srv1945627428:85 , srv1972172991:85 , srv1988586712:85 , srv2045020145:85 , srv2047285106:85 , srv2064842305:85 , srv2084281669:85 , srv220505498:85 , srv234253178:85 , srv289451626:85 , srv412692555:85 , srv421861027:85 , srv467517431:85 , srv522895109:85 , srv578800392:85 , srv612805008:85 , srv702862856:85 , srv922699842:85 , srv956387803:85 , srv105710637:86 , srv1295098778:86 , srv1300769512:86 , srv1321121208:86 , srv1642196904:86 , srv1658267909:86 , srv1676154559:86 , srv1863265544:86 , srv1987460246:86 , srv2029693782:86 , srv339937824:86 , srv51332168:86 , srv530119375:86 , srv628121863:86 , srv1827662603:87 , srv1203623539:617 }
2023-11-01 08:55:25,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-11-01 08:55:25,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv702862856=93, srv297472990=71, srv939444168=96, srv1246480938=13, srv628121863=92, srv612805008=88, srv615451105=89, srv141454514=21, srv1188472523=8, srv1741435981=43, srv1769062096=46, srv578800392=85, srv161851012=29, srv1119737969=7, srv987516475=98, srv1082807721=5, srv467517431=79, srv1658267909=35, srv2029693782=62, srv1737338140=42, srv1683949484=39, srv412692555=75, srv105710637=1, srv990680792=99, srv1107294494=6, srv1853812226=48, srv530119375=83, srv2084281669=66, srv1759036867=44, srv1067594788=2, srv1640410386=31, srv1666924533=37, srv1508337132=24, srv479408238=80, srv1353088442=19, srv420142783=76, srv1916313485=51, srv1321121208=18, srv1676154559=38, srv42961684=78, srv1018916932=0, srv1972172991=57, srv1612564290=28, srv622071650=90, srv1935519225=53, srv1940251433=54, srv755011886=94, srv1071119844=4, srv1643186708=33, srv1386779311=20, srv1300769512=17, srv339937824=72, srv922699842=95, srv1863265544=49, srv1987460246=58, srv1531710177=26, srv1763363853=45, srv289451626=70, srv1531027481=25, srv590495284=86, srv1945627428=55, srv51332168=81, srv956387803=97, srv38522392=74, srv421861027=77, srv598351471=87, srv120521674=11, srv142181724=22, srv1701261773=40, srv1533253444=27, srv626150897=91, srv1868202707=50, srv1205105054=10, srv1245137732=12, srv220505498=67, srv2004497819=61, srv1460588460=23, srv2045020145=63, srv1657212870=34, srv166041804=36, srv1988586712=59, srv1068973446=3, srv1295098778=16, srv1720981826=41, srv27978573=69, srv2047285106=64, srv2064842305=65, srv522895109=82, srv535400107=84, srv1274568401=14, srv1277157739=15, srv1642196904=32, srv1960891948=56, srv1990765450=60, srv1623941822=30, srv234253178=68, srv1917179960=52, srv1827662603=47, srv368808277=73, srv1203623539=9} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-11-01 08:55:25,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-11-01 08:55:25,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-11-01 08:55:25,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-11-01 08:55:25,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-11-01 08:55:25,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-11-01 08:55:25,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-11-01 08:55:25,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-11-01 08:55:25,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-11-01 08:55:25,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-11-01 08:55:25,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-11-01 08:55:25,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-11-01 08:55:25,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-11-01 08:55:25,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-11-01 08:55:25,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-11-01 08:55:25,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-11-01 08:55:25,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-11-01 08:55:25,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-11-01 08:55:25,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-11-01 08:55:25,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-11-01 08:55:25,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-11-01 08:55:25,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-11-01 08:55:25,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-11-01 08:55:25,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-11-01 08:55:25,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-11-01 08:55:25,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-11-01 08:55:25,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-11-01 08:55:25,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-11-01 08:55:25,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-11-01 08:55:25,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-11-01 08:55:25,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-11-01 08:55:25,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-11-01 08:55:25,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-11-01 08:55:25,300 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-11-01 08:55:25,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-11-01 08:55:25,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-11-01 08:55:25,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-11-01 08:55:25,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-11-01 08:55:25,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-11-01 08:55:25,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-11-01 08:55:25,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-11-01 08:55:25,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-11-01 08:55:25,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-11-01 08:55:25,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-11-01 08:55:25,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-11-01 08:55:25,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-11-01 08:55:25,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-11-01 08:55:25,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-11-01 08:55:25,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-11-01 08:55:25,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-11-01 08:55:25,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-11-01 08:55:25,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-11-01 08:55:25,301 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-11-01 08:55:25,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-11-01 08:55:25,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-11-01 08:55:25,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-11-01 08:55:25,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-11-01 08:55:25,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-11-01 08:55:25,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-11-01 08:55:25,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-11-01 08:55:25,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-11-01 08:55:25,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-11-01 08:55:25,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-11-01 08:55:25,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-11-01 08:55:25,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-11-01 08:55:25,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-11-01 08:55:25,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-11-01 08:55:25,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-11-01 08:55:25,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-11-01 08:55:25,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-11-01 08:55:25,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-11-01 08:55:25,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-11-01 08:55:25,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-11-01 08:55:25,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-11-01 08:55:25,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-11-01 08:55:25,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-11-01 08:55:25,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-11-01 08:55:25,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-11-01 08:55:25,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-11-01 08:55:25,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-11-01 08:55:25,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-11-01 08:55:25,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-11-01 08:55:25,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-11-01 08:55:25,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-11-01 08:55:25,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-11-01 08:55:25,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-11-01 08:55:25,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-11-01 08:55:25,303 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-11-01 08:55:25,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-11-01 08:55:25,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-11-01 08:55:25,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-11-01 08:55:25,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-11-01 08:55:25,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-11-01 08:55:25,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-11-01 08:55:25,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-11-01 08:55:25,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-11-01 08:55:25,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-11-01 08:55:25,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-11-01 08:55:25,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-11-01 08:55:25,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-11-01 08:55:25,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-11-01 08:55:25,304 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-11-01 08:55:25,410 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=1.7913508610245763E-5 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.4668347344853864E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018349396085439343); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.4668347344853864E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-01 08:55:25,411 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1082807721:83 , srv1246480938:83 , srv1460588460:83 , srv38522392:83 , srv987516475:83 , srv1018916932:84 , srv1119737969:84 , srv1188472523:84 , srv1245137732:84 , srv141454514:84 , srv1508337132:84 , srv1531710177:84 , srv1533253444:84 , srv1612564290:84 , srv161851012:84 , srv1640410386:84 , srv166041804:84 , srv1666924533:84 , srv1737338140:84 , srv1741435981:84 , srv1763363853:84 , srv1853812226:84 , srv1917179960:84 , srv1935519225:84 , srv1940251433:84 , srv1960891948:84 , srv1990765450:84 , srv2004497819:84 , srv27978573:84 , srv297472990:84 , srv368808277:84 , srv420142783:84 , srv42961684:84 , srv479408238:84 , srv535400107:84 , srv590495284:84 , srv598351471:84 , srv615451105:84 , srv622071650:84 , srv626150897:84 , srv755011886:84 , srv939444168:84 , srv990680792:84 , srv1067594788:85 , srv1068973446:85 , srv1071119844:85 , srv1107294494:85 , srv1205105054:85 , srv120521674:85 , srv1274568401:85 , srv1277157739:85 , srv1353088442:85 , srv1386779311:85 , srv142181724:85 , srv1531027481:85 , srv1623941822:85 , srv1643186708:85 , srv1657212870:85 , srv1683949484:85 , srv1701261773:85 , srv1720981826:85 , srv1759036867:85 , srv1769062096:85 , srv1868202707:85 , srv1916313485:85 , srv1945627428:85 , srv1972172991:85 , srv1988586712:85 , srv2045020145:85 , srv2047285106:85 , srv2064842305:85 , srv2084281669:85 , srv220505498:85 , srv234253178:85 , srv289451626:85 , srv412692555:85 , srv421861027:85 , srv467517431:85 , srv522895109:85 , srv578800392:85 , srv612805008:85 , srv702862856:85 , srv922699842:85 , srv956387803:85 , srv105710637:86 , srv1295098778:86 , srv1300769512:86 , srv1321121208:86 , srv1642196904:86 , srv1658267909:86 , srv1676154559:86 , srv1863265544:86 , srv1987460246:86 , srv2029693782:86 , srv339937824:86 , srv51332168:86 , srv530119375:86 , srv628121863:86 , srv1827662603:87 , srv1203623539:617 }
2023-11-01 08:55:25,438 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=139 (was 125) - SystemLoadAverage LEAK? -, ProcessCount=170 (was 170), AvailableMemoryMB=11024 (was 11517)
2023-11-01 08:55:25,445 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=139, ProcessCount=170, AvailableMemoryMB=11024
2023-11-01 08:55:25,446 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-11-01 08:55:25,446 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc.
2023-11-01 08:55:25,460 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1012507468:84 , srv11677802:84 , srv1197609474:84 , srv1220641947:84 , srv1237102490:84 , srv1481518187:84 , srv1491739805:84 , srv1512886788:84 , srv1686631592:84 , srv1707875409:84 , srv1714977005:84 , srv1787819368:84 , srv1815308941:84 , srv1815387422:84 , srv1898805980:84 , srv1905940930:84 , srv200062663:84 , srv2055760180:84 , srv2123480903:84 , srv271394121:84 , srv511605724:84 , srv54255867:84 , srv581950060:84 , srv707902580:84 , srv711476187:84 , srv75471321:84 , srv851933908:84 , srv950506151:84 , srv954647977:84 , srv1888734981:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90]
2023-11-01 08:55:25,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-11-01 08:55:25,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1481518187=5, srv271394121=20, srv711476187=25, srv1787819368=11, srv1707875409=9, srv54255867=22, srv1237102490=4, srv2123480903=19, srv851933908=27, srv11677802=1, srv1512886788=7, srv75471321=26, srv1714977005=10, srv200062663=17, srv707902580=24, srv950506151=28, srv1905940930=16, srv1898805980=15, srv1815387422=13, srv1686631592=8, srv511605724=21, srv2055760180=18, srv1197609474=2, srv1815308941=12, srv581950060=23, srv1888734981=14, srv1220641947=3, srv1012507468=0, srv954647977=29, srv1491739805=6} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-11-01 08:55:25,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-11-01 08:55:25,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-11-01 08:55:25,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-11-01 08:55:25,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-11-01 08:55:25,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-11-01 08:55:25,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-11-01 08:55:25,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-11-01 08:55:25,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-11-01 08:55:25,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-11-01 08:55:25,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-11-01 08:55:25,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-11-01 08:55:25,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-11-01 08:55:25,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-11-01 08:55:25,475 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-11-01 08:55:25,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-11-01 08:55:25,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-11-01 08:55:25,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-11-01 08:55:25,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-11-01 08:55:25,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-11-01 08:55:25,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-11-01 08:55:25,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-11-01 08:55:25,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-11-01 08:55:25,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-11-01 08:55:25,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-11-01 08:55:25,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-11-01 08:55:25,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-11-01 08:55:25,476 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-11-01 08:55:25,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-11-01 08:55:25,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-11-01 08:55:25,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-11-01 08:55:25,477 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-11-01 08:55:25,502 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.109086706085585, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-01 08:55:25,503 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912536771913147, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.109086706085585, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=64800000
2023-11-01 08:57:25,481 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 5622597 different iterations.  Found a solution that moves 1855 regions; Going from a computed imbalance of 0.9912536771913147 to a new imbalance of 4.329662539733032E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.687037037037037, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-01 08:57:25,487 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1012507468:90 , srv11677802:90 , srv1197609474:90 , srv1220641947:90 , srv1237102490:90 , srv1481518187:90 , srv1491739805:90 , srv1512886788:90 , srv1686631592:90 , srv1707875409:90 , srv1714977005:90 , srv1787819368:90 , srv1815308941:90 , srv1815387422:90 , srv1888734981:90 , srv1898805980:90 , srv1905940930:90 , srv200062663:90 , srv2055760180:90 , srv2123480903:90 , srv271394121:90 , srv511605724:90 , srv54255867:90 , srv581950060:90 , srv707902580:90 , srv711476187:90 , srv75471321:90 , srv851933908:90 , srv950506151:90 , srv954647977:90 }
2023-11-01 08:57:25,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-11-01 08:57:25,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1481518187=5, srv271394121=20, srv711476187=25, srv1787819368=11, srv1707875409=9, srv54255867=22, srv1237102490=4, srv2123480903=19, srv851933908=27, srv11677802=1, srv1512886788=7, srv75471321=26, srv1714977005=10, srv200062663=17, srv707902580=24, srv950506151=28, srv1905940930=16, srv1898805980=15, srv1815387422=13, srv1686631592=8, srv511605724=21, srv2055760180=18, srv1197609474=2, srv1815308941=12, srv581950060=23, srv1888734981=14, srv1220641947=3, srv1012507468=0, srv954647977=29, srv1491739805=6} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-11-01 08:57:25,491 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-11-01 08:57:25,491 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-11-01 08:57:25,491 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-11-01 08:57:25,491 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-11-01 08:57:25,491 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-11-01 08:57:25,491 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-11-01 08:57:25,491 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-11-01 08:57:25,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-11-01 08:57:25,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-11-01 08:57:25,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-11-01 08:57:25,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-11-01 08:57:25,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-11-01 08:57:25,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-11-01 08:57:25,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-11-01 08:57:25,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-11-01 08:57:25,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-11-01 08:57:25,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-11-01 08:57:25,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-11-01 08:57:25,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-11-01 08:57:25,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-11-01 08:57:25,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-11-01 08:57:25,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-11-01 08:57:25,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-11-01 08:57:25,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-11-01 08:57:25,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-11-01 08:57:25,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-11-01 08:57:25,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-11-01 08:57:25,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-11-01 08:57:25,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-11-01 08:57:25,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-11-01 08:57:25,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-11-01 08:57:25,504 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-01 08:57:25,505 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1012507468:90 , srv11677802:90 , srv1197609474:90 , srv1220641947:90 , srv1237102490:90 , srv1481518187:90 , srv1491739805:90 , srv1512886788:90 , srv1686631592:90 , srv1707875409:90 , srv1714977005:90 , srv1787819368:90 , srv1815308941:90 , srv1815387422:90 , srv1888734981:90 , srv1898805980:90 , srv1905940930:90 , srv200062663:90 , srv2055760180:90 , srv2123480903:90 , srv271394121:90 , srv511605724:90 , srv54255867:90 , srv581950060:90 , srv707902580:90 , srv711476187:90 , srv75471321:90 , srv851933908:90 , srv950506151:90 , srv954647977:90 }
2023-11-01 08:57:25,517 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=136 (was 139), ProcessCount=170 (was 170), AvailableMemoryMB=10822 (was 11024)