2024-03-29 22:53:17,602 INFO  [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins
2024-03-29 22:53:17,923 WARN  [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
2024-03-29 22:53:18,044 INFO  [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2024-03-29 22:53:18,111 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2024-03-29 22:53:18,119 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2024-03-29 22:53:18,135 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=285, ProcessCount=170, AvailableMemoryMB=3524
2024-03-29 22:53:18,137 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2024-03-29 22:53:18,138 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2024-03-29 22:53:18,259 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1005945545:84 , srv1025954206:84 , srv1044207349:84 , srv105525270:84 , srv1067887097:84 , srv1090943544:84 , srv109446742:84 , srv1102627528:84 , srv1103602659:84 , srv1112635943:84 , srv1163763608:84 , srv1180517111:84 , srv1182746634:84 , srv1189250858:84 , srv1250106600:84 , srv1252973397:84 , srv129963896:84 , srv1345749397:84 , srv135133702:84 , srv1374789206:84 , srv1381658917:84 , srv1408850334:84 , srv1434778561:84 , srv1449488821:84 , srv1472286763:84 , srv1486244807:84 , srv1493173141:84 , srv1515187509:84 , srv1539148314:84 , srv156920704:84 , srv1578128657:84 , srv1603505607:84 , srv1633044237:84 , srv1645612041:84 , srv1654454555:84 , srv165529755:84 , srv1655986937:84 , srv171214324:84 , srv1735741428:84 , srv1757445877:84 , srv1764740151:84 , srv1779241925:84 , srv1802583291:84 , srv1841398453:84 , srv1857758496:84 , srv1862784572:84 , srv1950961979:84 , srv1969703008:84 , srv2010735495:84 , srv2030957189:84 , srv2041814242:84 , srv2044543061:84 , srv2132121390:84 , srv23018263:84 , srv244542276:84 , srv247315713:84 , srv274673802:84 , srv303337637:84 , srv313571773:84 , srv388318348:84 , srv422063432:84 , srv440560632:84 , srv447286280:84 , srv454310868:84 , srv45977721:84 , srv465978652:84 , srv468791342:84 , srv47341800:84 , srv474005805:84 , srv474166278:84 , srv57157002:84 , srv588733703:84 , srv619939092:84 , srv637502859:84 , srv652518491:84 , srv656945728:84 , srv672180431:84 , srv67699350:84 , srv710544461:84 , srv717828340:84 , srv718091135:84 , srv720229892:84 , srv739336724:84 , srv75293789:84 , srv759864302:84 , srv782044236:84 , srv788528200:84 , srv80611940:84 , srv839036840:84 , srv857949307:84 , srv883510:84 , srv887156534:84 , srv888467728:84 , srv905353096:84 , srv915702657:84 , srv928353737:84 , srv961387986:84 , srv963630991:84 , srv968929201:84 , srv1717064651:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90]
2024-03-29 22:53:18,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-03-29 22:53:18,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1717064651=38, srv1539148314=28, srv928353737=96, srv720229892=82, srv474005805=69, srv171214324=37, srv75293789=84, srv1252973397=15, srv963630991=98, srv244542276=55, srv465978652=66, srv109446742=6, srv67699350=78, srv135133702=18, srv1381658917=20, srv1757445877=40, srv1857758496=45, srv129963896=16, srv888467728=93, srv839036840=89, srv1005945545=0, srv759864302=85, srv468791342=67, srv1735741428=39, srv454310868=64, srv1103602659=8, srv156920704=29, srv45977721=65, srv447286280=63, srv313571773=59, srv474166278=70, srv1025954206=1, srv1472286763=24, srv440560632=62, srv656945728=76, srv1633044237=32, srv1180517111=11, srv2030957189=50, srv717828340=80, srv1764740151=41, srv1841398453=44, srv961387986=97, srv968929201=99, srv1493173141=26, srv1779241925=42, srv1345749397=17, srv1189250858=13, srv637502859=74, srv1969703008=48, srv883510=91, srv739336724=83, srv57157002=71, srv588733703=72, srv652518491=75, srv105525270=3, srv1862784572=46, srv1102627528=7, srv710544461=79, srv1603505607=31, srv1163763608=10, srv1802583291=43, srv2041814242=51, srv915702657=95, srv80611940=88, srv1655986937=36, srv2044543061=52, srv1654454555=34, srv1449488821=23, srv1434778561=22, srv1044207349=2, srv1645612041=33, srv1515187509=27, srv47341800=68, srv619939092=73, srv1950961979=47, srv388318348=60, srv788528200=87, srv905353096=94, srv1112635943=9, srv422063432=61, srv782044236=86, srv887156534=92, srv857949307=90, srv23018263=54, srv2010735495=49, srv672180431=77, srv1486244807=25, srv1182746634=12, srv718091135=81, srv1250106600=14, srv247315713=56, srv303337637=58, srv1090943544=5, srv274673802=57, srv1578128657=30, srv2132121390=53, srv1067887097=4, srv1374789206=19, srv165529755=35, srv1408850334=21} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-03-29 22:53:18,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-03-29 22:53:18,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-03-29 22:53:18,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-03-29 22:53:18,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-03-29 22:53:18,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-03-29 22:53:18,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-03-29 22:53:18,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-03-29 22:53:18,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-03-29 22:53:18,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-03-29 22:53:18,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-03-29 22:53:18,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-03-29 22:53:18,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-03-29 22:53:18,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-03-29 22:53:18,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-03-29 22:53:18,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-03-29 22:53:18,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-03-29 22:53:18,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-03-29 22:53:18,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-03-29 22:53:18,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-03-29 22:53:18,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-03-29 22:53:18,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-03-29 22:53:18,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-03-29 22:53:18,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-03-29 22:53:18,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-03-29 22:53:18,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-03-29 22:53:18,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-03-29 22:53:18,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-03-29 22:53:18,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-03-29 22:53:18,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-03-29 22:53:18,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-03-29 22:53:18,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-03-29 22:53:18,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-03-29 22:53:18,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-03-29 22:53:18,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-03-29 22:53:18,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-03-29 22:53:18,378 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-03-29 22:53:18,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-03-29 22:53:18,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-03-29 22:53:18,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-03-29 22:53:18,379 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-03-29 22:53:18,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-03-29 22:53:18,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-03-29 22:53:18,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-03-29 22:53:18,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-03-29 22:53:18,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-03-29 22:53:18,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-03-29 22:53:18,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-03-29 22:53:18,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-03-29 22:53:18,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-03-29 22:53:18,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-03-29 22:53:18,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-03-29 22:53:18,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-03-29 22:53:18,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-03-29 22:53:18,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-03-29 22:53:18,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-03-29 22:53:18,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-03-29 22:53:18,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-03-29 22:53:18,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-03-29 22:53:18,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-03-29 22:53:18,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-03-29 22:53:18,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-03-29 22:53:18,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-03-29 22:53:18,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-03-29 22:53:18,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-03-29 22:53:18,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-03-29 22:53:18,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-03-29 22:53:18,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-03-29 22:53:18,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-03-29 22:53:18,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-03-29 22:53:18,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-03-29 22:53:18,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-03-29 22:53:18,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-03-29 22:53:18,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-03-29 22:53:18,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-03-29 22:53:18,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-03-29 22:53:18,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-03-29 22:53:18,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-03-29 22:53:18,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-03-29 22:53:18,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-03-29 22:53:18,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-03-29 22:53:18,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-03-29 22:53:18,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-03-29 22:53:18,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-03-29 22:53:18,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-03-29 22:53:18,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-03-29 22:53:18,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-03-29 22:53:18,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-03-29 22:53:18,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-03-29 22:53:18,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-03-29 22:53:18,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-03-29 22:53:18,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-03-29 22:53:18,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-03-29 22:53:18,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-03-29 22:53:18,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-03-29 22:53:18,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-03-29 22:53:18,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-03-29 22:53:18,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-03-29 22:53:18,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-03-29 22:53:18,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-03-29 22:53:18,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-03-29 22:53:18,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-03-29 22:53:18,739 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-29 22:53:18,739 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-03-29 22:53:18,739 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-03-29 22:53:19,163 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 739 ms to try 5000 different iterations.  Found a solution that moves 4653 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.18086632812554618. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.040629784628121605, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05946638885049326, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.517, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.040629784628121605, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17625, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.241, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-29 22:53:19,183 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv135133702:80 , srv1486244807:81 , srv1735741428:81 , srv710544461:81 , srv57157002:82 , srv720229892:82 , srv1090943544:83 , srv1163763608:83 , srv1449488821:83 , srv1603505607:83 , srv244542276:83 , srv303337637:83 , srv47341800:83 , srv968929201:83 , srv1112635943:84 , srv129963896:84 , srv1434778561:84 , srv1578128657:84 , srv1757445877:84 , srv388318348:84 , srv447286280:84 , srv619939092:84 , srv637502859:84 , srv80611940:84 , srv905353096:84 , srv1005945545:85 , srv1067887097:85 , srv1102627528:85 , srv1764740151:85 , srv1841398453:85 , srv1969703008:85 , srv2030957189:85 , srv2132121390:85 , srv422063432:85 , srv454310868:85 , srv45977721:85 , srv474005805:85 , srv75293789:85 , srv915702657:85 , srv105525270:86 , srv1103602659:86 , srv1250106600:86 , srv1252973397:86 , srv1654454555:86 , srv313571773:86 , srv465978652:86 , srv67699350:86 , srv739336724:86 , srv759864302:86 , srv839036840:86 , srv928353737:86 , srv961387986:86 , srv1044207349:87 , srv109446742:87 , srv1408850334:87 , srv1493173141:87 , srv156920704:87 , srv1633044237:87 , srv1645612041:87 , srv171214324:87 , srv2010735495:87 , srv247315713:87 , srv274673802:87 , srv672180431:87 , srv718091135:87 , srv788528200:87 , srv883510:87 , srv1025954206:88 , srv1189250858:88 , srv1345749397:88 , srv1374789206:88 , srv1381658917:88 , srv1515187509:88 , srv165529755:88 , srv468791342:88 , srv474166278:88 , srv652518491:88 , srv656945728:88 , srv857949307:88 , srv1180517111:89 , srv1182746634:89 , srv1779241925:89 , srv2044543061:89 , srv23018263:89 , srv440560632:89 , srv888467728:89 , srv963630991:89 , srv1539148314:90 , srv1802583291:90 , srv1857758496:90 , srv1950961979:90 , srv588733703:90 , srv887156534:90 , srv1472286763:91 , srv2041814242:91 , srv717828340:91 , srv1655986937:92 , srv782044236:92 , srv1862784572:96 , srv1717064651:451 }
2024-03-29 22:53:19,188 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-03-29 22:53:19,189 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1717064651=38, srv1539148314=28, srv928353737=96, srv720229892=82, srv474005805=69, srv171214324=37, srv75293789=84, srv1252973397=15, srv963630991=98, srv244542276=55, srv465978652=66, srv109446742=6, srv67699350=78, srv135133702=18, srv1381658917=20, srv1757445877=40, srv1857758496=45, srv129963896=16, srv888467728=93, srv839036840=89, srv1005945545=0, srv759864302=85, srv468791342=67, srv1735741428=39, srv454310868=64, srv1103602659=8, srv156920704=29, srv45977721=65, srv447286280=63, srv313571773=59, srv474166278=70, srv1025954206=1, srv1472286763=24, srv440560632=62, srv656945728=76, srv1633044237=32, srv1180517111=11, srv2030957189=50, srv717828340=80, srv1764740151=41, srv1841398453=44, srv961387986=97, srv968929201=99, srv1493173141=26, srv1779241925=42, srv1345749397=17, srv1189250858=13, srv637502859=74, srv1969703008=48, srv883510=91, srv739336724=83, srv57157002=71, srv588733703=72, srv652518491=75, srv105525270=3, srv1862784572=46, srv1102627528=7, srv710544461=79, srv1603505607=31, srv1163763608=10, srv1802583291=43, srv2041814242=51, srv915702657=95, srv80611940=88, srv1655986937=36, srv2044543061=52, srv1654454555=34, srv1449488821=23, srv1434778561=22, srv1044207349=2, srv1645612041=33, srv1515187509=27, srv47341800=68, srv619939092=73, srv1950961979=47, srv388318348=60, srv788528200=87, srv905353096=94, srv1112635943=9, srv422063432=61, srv782044236=86, srv887156534=92, srv857949307=90, srv23018263=54, srv2010735495=49, srv672180431=77, srv1486244807=25, srv1182746634=12, srv718091135=81, srv1250106600=14, srv247315713=56, srv303337637=58, srv1090943544=5, srv274673802=57, srv1578128657=30, srv2132121390=53, srv1067887097=4, srv1374789206=19, srv165529755=35, srv1408850334=21} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-03-29 22:53:19,209 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-03-29 22:53:19,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-03-29 22:53:19,210 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-03-29 22:53:19,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-03-29 22:53:19,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-03-29 22:53:19,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-03-29 22:53:19,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-03-29 22:53:19,211 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-03-29 22:53:19,212 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-03-29 22:53:19,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-03-29 22:53:19,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-03-29 22:53:19,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-03-29 22:53:19,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-03-29 22:53:19,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-03-29 22:53:19,213 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-03-29 22:53:19,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-03-29 22:53:19,214 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-03-29 22:53:19,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-03-29 22:53:19,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-03-29 22:53:19,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-03-29 22:53:19,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-03-29 22:53:19,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-03-29 22:53:19,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-03-29 22:53:19,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-03-29 22:53:19,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-03-29 22:53:19,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-03-29 22:53:19,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-03-29 22:53:19,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-03-29 22:53:19,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-03-29 22:53:19,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-03-29 22:53:19,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-03-29 22:53:19,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-03-29 22:53:19,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-03-29 22:53:19,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-03-29 22:53:19,216 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-03-29 22:53:19,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-03-29 22:53:19,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-03-29 22:53:19,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-03-29 22:53:19,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-03-29 22:53:19,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-03-29 22:53:19,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-03-29 22:53:19,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-03-29 22:53:19,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-03-29 22:53:19,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-03-29 22:53:19,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-03-29 22:53:19,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-03-29 22:53:19,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-03-29 22:53:19,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-03-29 22:53:19,217 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-03-29 22:53:19,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-03-29 22:53:19,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-03-29 22:53:19,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-03-29 22:53:19,218 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-03-29 22:53:19,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-03-29 22:53:19,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-03-29 22:53:19,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-03-29 22:53:19,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-03-29 22:53:19,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-03-29 22:53:19,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-03-29 22:53:19,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-03-29 22:53:19,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-03-29 22:53:19,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-03-29 22:53:19,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-03-29 22:53:19,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-03-29 22:53:19,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-03-29 22:53:19,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-03-29 22:53:19,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-03-29 22:53:19,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-03-29 22:53:19,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-03-29 22:53:19,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-03-29 22:53:19,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-03-29 22:53:19,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-03-29 22:53:19,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-03-29 22:53:19,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-03-29 22:53:19,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-03-29 22:53:19,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-03-29 22:53:19,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-03-29 22:53:19,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-03-29 22:53:19,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-03-29 22:53:19,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-03-29 22:53:19,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-03-29 22:53:19,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-03-29 22:53:19,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-03-29 22:53:19,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-03-29 22:53:19,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-03-29 22:53:19,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-03-29 22:53:19,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-03-29 22:53:19,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-03-29 22:53:19,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-03-29 22:53:19,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-03-29 22:53:19,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-03-29 22:53:19,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-03-29 22:53:19,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-03-29 22:53:19,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-03-29 22:53:19,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-03-29 22:53:19,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-03-29 22:53:19,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-03-29 22:53:19,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-03-29 22:53:19,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-03-29 22:53:19,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-03-29 22:53:19,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-03-29 22:53:19,403 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.040629784628121605, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05946638885049326, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.040629784628121605, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17625, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.241, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-29 22:53:19,403 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-03-29 22:53:19,404 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.18083374712317843, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.040629784628121605, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05946638885049326, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.040629784628121605, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17625, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.241, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-03-29 22:53:19,764 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 503 ms to try 5000 different iterations.  Found a solution that moves 2665 regions; Going from a computed imbalance of 0.18083374712317843 to a new imbalance of 0.016015558473024133. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02138023277260212); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.035897972393963355, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.2961111111111111, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02138023277260212); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012833333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04641666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-29 22:53:19,776 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1862784572:78 , srv1472286763:81 , srv1539148314:81 , srv1180517111:82 , srv1182746634:82 , srv1345749397:82 , srv165529755:82 , srv1802583291:82 , srv2041814242:82 , srv2044543061:82 , srv23018263:82 , srv440560632:82 , srv588733703:82 , srv717828340:82 , srv888467728:82 , srv109446742:83 , srv1189250858:83 , srv1374789206:83 , srv1381658917:83 , srv1779241925:83 , srv1857758496:83 , srv1950961979:83 , srv782044236:83 , srv788528200:83 , srv887156534:83 , srv963630991:83 , srv1633044237:84 , srv1645612041:84 , srv1655986937:84 , srv2010735495:84 , srv652518491:84 , srv1025954206:85 , srv1103602659:85 , srv156920704:85 , srv1654454555:85 , srv171214324:85 , srv465978652:85 , srv474166278:85 , srv656945728:85 , srv718091135:85 , srv928353737:85 , srv1044207349:86 , srv1102627528:86 , srv1250106600:86 , srv1252973397:86 , srv129963896:86 , srv1434778561:86 , srv1764740151:86 , srv1969703008:86 , srv2030957189:86 , srv2132121390:86 , srv247315713:86 , srv313571773:86 , srv672180431:86 , srv75293789:86 , srv857949307:86 , srv905353096:86 , srv1005945545:87 , srv1067887097:87 , srv1090943544:87 , srv1408850334:87 , srv1515187509:87 , srv1603505607:87 , srv274673802:87 , srv422063432:87 , srv454310868:87 , srv45977721:87 , srv468791342:87 , srv67699350:87 , srv739336724:87 , srv839036840:87 , srv883510:87 , srv961387986:87 , srv968929201:87 , srv1112635943:88 , srv1449488821:88 , srv1493173141:88 , srv1841398453:88 , srv244542276:88 , srv388318348:88 , srv474005805:88 , srv619939092:88 , srv637502859:88 , srv80611940:88 , srv915702657:88 , srv105525270:89 , srv1578128657:89 , srv1757445877:89 , srv447286280:89 , srv47341800:89 , srv57157002:89 , srv720229892:89 , srv1735741428:90 , srv303337637:90 , srv759864302:90 , srv1163763608:91 , srv135133702:91 , srv1486244807:91 , srv710544461:91 , srv1717064651:513 }
2024-03-29 22:53:19,783 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-03-29 22:53:19,784 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1717064651=38, srv1539148314=28, srv928353737=96, srv720229892=82, srv474005805=69, srv171214324=37, srv75293789=84, srv1252973397=15, srv963630991=98, srv244542276=55, srv465978652=66, srv109446742=6, srv67699350=78, srv135133702=18, srv1381658917=20, srv1757445877=40, srv1857758496=45, srv129963896=16, srv888467728=93, srv839036840=89, srv1005945545=0, srv759864302=85, srv468791342=67, srv1735741428=39, srv454310868=64, srv1103602659=8, srv156920704=29, srv45977721=65, srv447286280=63, srv313571773=59, srv474166278=70, srv1025954206=1, srv1472286763=24, srv440560632=62, srv656945728=76, srv1633044237=32, srv1180517111=11, srv2030957189=50, srv717828340=80, srv1764740151=41, srv1841398453=44, srv961387986=97, srv968929201=99, srv1493173141=26, srv1779241925=42, srv1345749397=17, srv1189250858=13, srv637502859=74, srv1969703008=48, srv883510=91, srv739336724=83, srv57157002=71, srv588733703=72, srv652518491=75, srv105525270=3, srv1862784572=46, srv1102627528=7, srv710544461=79, srv1603505607=31, srv1163763608=10, srv1802583291=43, srv2041814242=51, srv915702657=95, srv80611940=88, srv1655986937=36, srv2044543061=52, srv1654454555=34, srv1449488821=23, srv1434778561=22, srv1044207349=2, srv1645612041=33, srv1515187509=27, srv47341800=68, srv619939092=73, srv1950961979=47, srv388318348=60, srv788528200=87, srv905353096=94, srv1112635943=9, srv422063432=61, srv782044236=86, srv887156534=92, srv857949307=90, srv23018263=54, srv2010735495=49, srv672180431=77, srv1486244807=25, srv1182746634=12, srv718091135=81, srv1250106600=14, srv247315713=56, srv303337637=58, srv1090943544=5, srv274673802=57, srv1578128657=30, srv2132121390=53, srv1067887097=4, srv1374789206=19, srv165529755=35, srv1408850334=21} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-03-29 22:53:19,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-03-29 22:53:19,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-03-29 22:53:19,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-03-29 22:53:19,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-03-29 22:53:19,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-03-29 22:53:19,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-03-29 22:53:19,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-03-29 22:53:19,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-03-29 22:53:19,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-03-29 22:53:19,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-03-29 22:53:19,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-03-29 22:53:19,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-03-29 22:53:19,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-03-29 22:53:19,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-03-29 22:53:19,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-03-29 22:53:19,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-03-29 22:53:19,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-03-29 22:53:19,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-03-29 22:53:19,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-03-29 22:53:19,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-03-29 22:53:19,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-03-29 22:53:19,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-03-29 22:53:19,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-03-29 22:53:19,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-03-29 22:53:19,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-03-29 22:53:19,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-03-29 22:53:19,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-03-29 22:53:19,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-03-29 22:53:19,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-03-29 22:53:19,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-03-29 22:53:19,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-03-29 22:53:19,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-03-29 22:53:19,826 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-03-29 22:53:19,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-03-29 22:53:19,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-03-29 22:53:19,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-03-29 22:53:19,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-03-29 22:53:19,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-03-29 22:53:19,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-03-29 22:53:19,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-03-29 22:53:19,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-03-29 22:53:19,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-03-29 22:53:19,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-03-29 22:53:19,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-03-29 22:53:19,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-03-29 22:53:19,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-03-29 22:53:19,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-03-29 22:53:19,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-03-29 22:53:19,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-03-29 22:53:19,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-03-29 22:53:19,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-03-29 22:53:19,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-03-29 22:53:19,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-03-29 22:53:19,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-03-29 22:53:19,828 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-03-29 22:53:19,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-03-29 22:53:19,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-03-29 22:53:19,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-03-29 22:53:19,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-03-29 22:53:19,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-03-29 22:53:19,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-03-29 22:53:19,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-03-29 22:53:19,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-03-29 22:53:19,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-03-29 22:53:19,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-03-29 22:53:19,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-03-29 22:53:19,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-03-29 22:53:19,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-03-29 22:53:19,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-03-29 22:53:19,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-03-29 22:53:19,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-03-29 22:53:19,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-03-29 22:53:19,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-03-29 22:53:19,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-03-29 22:53:19,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-03-29 22:53:19,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-03-29 22:53:19,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-03-29 22:53:19,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-03-29 22:53:19,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-03-29 22:53:19,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-03-29 22:53:19,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-03-29 22:53:19,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-03-29 22:53:19,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-03-29 22:53:19,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-03-29 22:53:19,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-03-29 22:53:19,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-03-29 22:53:19,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-03-29 22:53:19,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-03-29 22:53:19,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-03-29 22:53:19,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-03-29 22:53:19,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-03-29 22:53:19,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-03-29 22:53:19,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-03-29 22:53:19,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-03-29 22:53:19,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-03-29 22:53:19,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-03-29 22:53:19,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-03-29 22:53:19,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-03-29 22:53:19,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-03-29 22:53:19,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-03-29 22:53:19,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-03-29 22:53:19,996 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02138023277260212); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.035897972393963355, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02138023277260212); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012833333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04641666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-29 22:53:19,997 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-03-29 22:53:19,997 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.01599689774418038, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02138023277260212); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.035897972393963355, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02138023277260212); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012833333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04641666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-03-29 22:53:20,387 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 535 ms to try 5000 different iterations.  Found a solution that moves 1003 regions; Going from a computed imbalance of 0.01599689774418038 to a new imbalance of 7.420361558410017E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.004533342135947968); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00843540185701534); MoveCostFunction : (multiplier=7.0, imbalance=0.11144444444444444, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.004533342135947968); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004166666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-29 22:53:20,390 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv105525270:82 , srv1655986937:82 , srv759864302:82 , srv782044236:82 , srv1163763608:83 , srv1493173141:83 , srv1515187509:83 , srv468791342:83 , srv1025954206:84 , srv1044207349:84 , srv1408850334:84 , srv1757445877:84 , srv1841398453:84 , srv1862784572:84 , srv274673802:84 , srv474005805:84 , srv672180431:84 , srv717828340:84 , srv857949307:84 , srv883510:84 , srv887156534:84 , srv1005945545:85 , srv1250106600:85 , srv1252973397:85 , srv1472286763:85 , srv1486244807:85 , srv156920704:85 , srv1578128657:85 , srv1779241925:85 , srv1857758496:85 , srv1950961979:85 , srv2041814242:85 , srv247315713:85 , srv303337637:85 , srv388318348:85 , srv422063432:85 , srv447286280:85 , srv454310868:85 , srv45977721:85 , srv474166278:85 , srv588733703:85 , srv619939092:85 , srv637502859:85 , srv656945728:85 , srv739336724:85 , srv839036840:85 , srv915702657:85 , srv961387986:85 , srv963630991:85 , srv1067887097:86 , srv1103602659:86 , srv1112635943:86 , srv1182746634:86 , srv1189250858:86 , srv135133702:86 , srv1449488821:86 , srv1645612041:86 , srv171214324:86 , srv2030957189:86 , srv2132121390:86 , srv23018263:86 , srv244542276:86 , srv313571773:86 , srv465978652:86 , srv47341800:86 , srv652518491:86 , srv67699350:86 , srv710544461:86 , srv720229892:86 , srv928353737:86 , srv1090943544:87 , srv109446742:87 , srv1102627528:87 , srv1180517111:87 , srv1345749397:87 , srv1374789206:87 , srv1381658917:87 , srv1434778561:87 , srv1539148314:87 , srv1633044237:87 , srv1654454555:87 , srv1735741428:87 , srv1764740151:87 , srv1802583291:87 , srv1969703008:87 , srv2010735495:87 , srv2044543061:87 , srv440560632:87 , srv57157002:87 , srv718091135:87 , srv75293789:87 , srv80611940:87 , srv888467728:87 , srv905353096:87 , srv129963896:88 , srv1603505607:88 , srv165529755:88 , srv788528200:88 , srv968929201:88 , srv1717064651:534 }
2024-03-29 22:53:20,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-03-29 22:53:20,397 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1717064651=38, srv1539148314=28, srv928353737=96, srv720229892=82, srv474005805=69, srv171214324=37, srv75293789=84, srv1252973397=15, srv963630991=98, srv244542276=55, srv465978652=66, srv109446742=6, srv67699350=78, srv135133702=18, srv1381658917=20, srv1757445877=40, srv1857758496=45, srv129963896=16, srv888467728=93, srv839036840=89, srv1005945545=0, srv759864302=85, srv468791342=67, srv1735741428=39, srv454310868=64, srv1103602659=8, srv156920704=29, srv45977721=65, srv447286280=63, srv313571773=59, srv474166278=70, srv1025954206=1, srv1472286763=24, srv440560632=62, srv656945728=76, srv1633044237=32, srv1180517111=11, srv2030957189=50, srv717828340=80, srv1764740151=41, srv1841398453=44, srv961387986=97, srv968929201=99, srv1493173141=26, srv1779241925=42, srv1345749397=17, srv1189250858=13, srv637502859=74, srv1969703008=48, srv883510=91, srv739336724=83, srv57157002=71, srv588733703=72, srv652518491=75, srv105525270=3, srv1862784572=46, srv1102627528=7, srv710544461=79, srv1603505607=31, srv1163763608=10, srv1802583291=43, srv2041814242=51, srv915702657=95, srv80611940=88, srv1655986937=36, srv2044543061=52, srv1654454555=34, srv1449488821=23, srv1434778561=22, srv1044207349=2, srv1645612041=33, srv1515187509=27, srv47341800=68, srv619939092=73, srv1950961979=47, srv388318348=60, srv788528200=87, srv905353096=94, srv1112635943=9, srv422063432=61, srv782044236=86, srv887156534=92, srv857949307=90, srv23018263=54, srv2010735495=49, srv672180431=77, srv1486244807=25, srv1182746634=12, srv718091135=81, srv1250106600=14, srv247315713=56, srv303337637=58, srv1090943544=5, srv274673802=57, srv1578128657=30, srv2132121390=53, srv1067887097=4, srv1374789206=19, srv165529755=35, srv1408850334=21} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-03-29 22:53:20,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-03-29 22:53:20,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-03-29 22:53:20,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-03-29 22:53:20,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-03-29 22:53:20,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-03-29 22:53:20,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-03-29 22:53:20,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-03-29 22:53:20,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-03-29 22:53:20,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-03-29 22:53:20,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-03-29 22:53:20,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-03-29 22:53:20,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-03-29 22:53:20,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-03-29 22:53:20,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-03-29 22:53:20,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-03-29 22:53:20,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-03-29 22:53:20,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-03-29 22:53:20,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-03-29 22:53:20,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-03-29 22:53:20,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-03-29 22:53:20,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-03-29 22:53:20,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-03-29 22:53:20,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-03-29 22:53:20,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-03-29 22:53:20,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-03-29 22:53:20,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-03-29 22:53:20,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-03-29 22:53:20,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-03-29 22:53:20,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-03-29 22:53:20,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-03-29 22:53:20,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-03-29 22:53:20,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-03-29 22:53:20,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-03-29 22:53:20,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-03-29 22:53:20,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-03-29 22:53:20,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-03-29 22:53:20,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-03-29 22:53:20,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-03-29 22:53:20,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-03-29 22:53:20,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-03-29 22:53:20,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-03-29 22:53:20,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-03-29 22:53:20,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-03-29 22:53:20,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-03-29 22:53:20,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-03-29 22:53:20,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-03-29 22:53:20,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-03-29 22:53:20,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-03-29 22:53:20,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-03-29 22:53:20,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-03-29 22:53:20,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-03-29 22:53:20,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-03-29 22:53:20,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-03-29 22:53:20,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-03-29 22:53:20,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-03-29 22:53:20,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-03-29 22:53:20,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-03-29 22:53:20,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-03-29 22:53:20,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-03-29 22:53:20,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-03-29 22:53:20,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-03-29 22:53:20,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-03-29 22:53:20,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-03-29 22:53:20,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-03-29 22:53:20,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-03-29 22:53:20,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-03-29 22:53:20,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-03-29 22:53:20,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-03-29 22:53:20,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-03-29 22:53:20,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-03-29 22:53:20,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-03-29 22:53:20,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-03-29 22:53:20,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-03-29 22:53:20,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-03-29 22:53:20,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-03-29 22:53:20,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-03-29 22:53:20,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-03-29 22:53:20,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-03-29 22:53:20,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-03-29 22:53:20,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-03-29 22:53:20,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-03-29 22:53:20,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-03-29 22:53:20,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-03-29 22:53:20,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-03-29 22:53:20,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-03-29 22:53:20,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-03-29 22:53:20,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-03-29 22:53:20,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-03-29 22:53:20,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-03-29 22:53:20,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-03-29 22:53:20,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-03-29 22:53:20,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-03-29 22:53:20,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-03-29 22:53:20,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-03-29 22:53:20,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-03-29 22:53:20,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-03-29 22:53:20,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-03-29 22:53:20,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-03-29 22:53:20,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-03-29 22:53:20,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-03-29 22:53:20,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-03-29 22:53:20,588 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.004533342135947968); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00843540185701534); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.004533342135947968); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004166666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-29 22:53:20,588 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-03-29 22:53:20,589 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=7.350129997320763E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.004533342135947968); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00843540185701534); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.004533342135947968); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004166666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-03-29 22:53:21,033 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 587 ms to try 5000 different iterations.  Found a solution that moves 244 regions; Going from a computed imbalance of 7.350129997320763E-4 to a new imbalance of 2.076087081331231E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.178341294946416E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.002118805753879094); MoveCostFunction : (multiplier=7.0, imbalance=0.02711111111111111, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.178341294946416E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-29 22:53:21,034 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1802583291:82 , srv1654454555:83 , srv67699350:83 , srv718091135:83 , srv80611940:83 , srv1067887097:84 , srv109446742:84 , srv1102627528:84 , srv1112635943:84 , srv1163763608:84 , srv1180517111:84 , srv1189250858:84 , srv129963896:84 , srv1374789206:84 , srv1381658917:84 , srv1408850334:84 , srv1493173141:84 , srv1515187509:84 , srv1539148314:84 , srv1578128657:84 , srv1603505607:84 , srv1633044237:84 , srv165529755:84 , srv1655986937:84 , srv171214324:84 , srv1735741428:84 , srv1764740151:84 , srv1857758496:84 , srv1862784572:84 , srv1969703008:84 , srv2010735495:84 , srv2030957189:84 , srv2041814242:84 , srv247315713:84 , srv274673802:84 , srv303337637:84 , srv313571773:84 , srv388318348:84 , srv440560632:84 , srv447286280:84 , srv468791342:84 , srv47341800:84 , srv474166278:84 , srv652518491:84 , srv656945728:84 , srv710544461:84 , srv739336724:84 , srv75293789:84 , srv759864302:84 , srv788528200:84 , srv839036840:84 , srv857949307:84 , srv888467728:84 , srv915702657:84 , srv961387986:84 , srv968929201:84 , srv1005945545:85 , srv1025954206:85 , srv1044207349:85 , srv105525270:85 , srv1090943544:85 , srv1103602659:85 , srv1182746634:85 , srv1250106600:85 , srv1252973397:85 , srv1345749397:85 , srv135133702:85 , srv1434778561:85 , srv1449488821:85 , srv1472286763:85 , srv1486244807:85 , srv156920704:85 , srv1645612041:85 , srv1757445877:85 , srv1779241925:85 , srv1841398453:85 , srv1950961979:85 , srv2044543061:85 , srv2132121390:85 , srv23018263:85 , srv244542276:85 , srv45977721:85 , srv465978652:85 , srv474005805:85 , srv57157002:85 , srv588733703:85 , srv619939092:85 , srv637502859:85 , srv672180431:85 , srv717828340:85 , srv720229892:85 , srv782044236:85 , srv883510:85 , srv887156534:85 , srv905353096:85 , srv963630991:85 , srv422063432:86 , srv454310868:86 , srv928353737:86 , srv1717064651:644 }
2024-03-29 22:53:21,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-03-29 22:53:21,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1717064651=38, srv1539148314=28, srv928353737=96, srv720229892=82, srv474005805=69, srv171214324=37, srv75293789=84, srv1252973397=15, srv963630991=98, srv244542276=55, srv465978652=66, srv109446742=6, srv67699350=78, srv135133702=18, srv1381658917=20, srv1757445877=40, srv1857758496=45, srv129963896=16, srv888467728=93, srv839036840=89, srv1005945545=0, srv759864302=85, srv468791342=67, srv1735741428=39, srv454310868=64, srv1103602659=8, srv156920704=29, srv45977721=65, srv447286280=63, srv313571773=59, srv474166278=70, srv1025954206=1, srv1472286763=24, srv440560632=62, srv656945728=76, srv1633044237=32, srv1180517111=11, srv2030957189=50, srv717828340=80, srv1764740151=41, srv1841398453=44, srv961387986=97, srv968929201=99, srv1493173141=26, srv1779241925=42, srv1345749397=17, srv1189250858=13, srv637502859=74, srv1969703008=48, srv883510=91, srv739336724=83, srv57157002=71, srv588733703=72, srv652518491=75, srv105525270=3, srv1862784572=46, srv1102627528=7, srv710544461=79, srv1603505607=31, srv1163763608=10, srv1802583291=43, srv2041814242=51, srv915702657=95, srv80611940=88, srv1655986937=36, srv2044543061=52, srv1654454555=34, srv1449488821=23, srv1434778561=22, srv1044207349=2, srv1645612041=33, srv1515187509=27, srv47341800=68, srv619939092=73, srv1950961979=47, srv388318348=60, srv788528200=87, srv905353096=94, srv1112635943=9, srv422063432=61, srv782044236=86, srv887156534=92, srv857949307=90, srv23018263=54, srv2010735495=49, srv672180431=77, srv1486244807=25, srv1182746634=12, srv718091135=81, srv1250106600=14, srv247315713=56, srv303337637=58, srv1090943544=5, srv274673802=57, srv1578128657=30, srv2132121390=53, srv1067887097=4, srv1374789206=19, srv165529755=35, srv1408850334=21} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-03-29 22:53:21,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-03-29 22:53:21,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-03-29 22:53:21,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-03-29 22:53:21,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-03-29 22:53:21,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-03-29 22:53:21,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-03-29 22:53:21,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-03-29 22:53:21,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-03-29 22:53:21,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-03-29 22:53:21,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-03-29 22:53:21,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-03-29 22:53:21,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-03-29 22:53:21,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-03-29 22:53:21,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-03-29 22:53:21,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-03-29 22:53:21,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-03-29 22:53:21,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-03-29 22:53:21,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-03-29 22:53:21,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-03-29 22:53:21,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-03-29 22:53:21,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-03-29 22:53:21,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-03-29 22:53:21,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-03-29 22:53:21,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-03-29 22:53:21,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-03-29 22:53:21,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-03-29 22:53:21,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-03-29 22:53:21,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-03-29 22:53:21,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-03-29 22:53:21,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-03-29 22:53:21,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-03-29 22:53:21,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-03-29 22:53:21,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-03-29 22:53:21,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-03-29 22:53:21,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-03-29 22:53:21,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-03-29 22:53:21,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-03-29 22:53:21,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-03-29 22:53:21,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-03-29 22:53:21,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-03-29 22:53:21,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-03-29 22:53:21,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-03-29 22:53:21,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-03-29 22:53:21,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-03-29 22:53:21,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-03-29 22:53:21,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-03-29 22:53:21,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-03-29 22:53:21,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-03-29 22:53:21,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-03-29 22:53:21,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-03-29 22:53:21,051 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-03-29 22:53:21,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-03-29 22:53:21,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-03-29 22:53:21,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-03-29 22:53:21,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-03-29 22:53:21,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-03-29 22:53:21,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-03-29 22:53:21,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-03-29 22:53:21,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-03-29 22:53:21,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-03-29 22:53:21,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-03-29 22:53:21,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-03-29 22:53:21,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-03-29 22:53:21,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-03-29 22:53:21,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-03-29 22:53:21,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-03-29 22:53:21,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-03-29 22:53:21,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-03-29 22:53:21,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-03-29 22:53:21,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-03-29 22:53:21,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-03-29 22:53:21,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-03-29 22:53:21,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-03-29 22:53:21,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-03-29 22:53:21,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-03-29 22:53:21,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-03-29 22:53:21,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-03-29 22:53:21,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-03-29 22:53:21,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-03-29 22:53:21,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-03-29 22:53:21,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-03-29 22:53:21,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-03-29 22:53:21,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-03-29 22:53:21,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-03-29 22:53:21,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-03-29 22:53:21,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-03-29 22:53:21,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-03-29 22:53:21,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-03-29 22:53:21,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-03-29 22:53:21,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-03-29 22:53:21,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-03-29 22:53:21,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-03-29 22:53:21,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-03-29 22:53:21,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-03-29 22:53:21,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-03-29 22:53:21,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-03-29 22:53:21,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-03-29 22:53:21,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-03-29 22:53:21,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-03-29 22:53:21,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-03-29 22:53:21,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-03-29 22:53:21,159 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=1.9052346296285582E-5 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.178341294946416E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.002118805753879094); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.178341294946416E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-29 22:53:21,159 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1802583291:82 , srv1654454555:83 , srv67699350:83 , srv718091135:83 , srv80611940:83 , srv1067887097:84 , srv109446742:84 , srv1102627528:84 , srv1112635943:84 , srv1163763608:84 , srv1180517111:84 , srv1189250858:84 , srv129963896:84 , srv1374789206:84 , srv1381658917:84 , srv1408850334:84 , srv1493173141:84 , srv1515187509:84 , srv1539148314:84 , srv1578128657:84 , srv1603505607:84 , srv1633044237:84 , srv165529755:84 , srv1655986937:84 , srv171214324:84 , srv1735741428:84 , srv1764740151:84 , srv1857758496:84 , srv1862784572:84 , srv1969703008:84 , srv2010735495:84 , srv2030957189:84 , srv2041814242:84 , srv247315713:84 , srv274673802:84 , srv303337637:84 , srv313571773:84 , srv388318348:84 , srv440560632:84 , srv447286280:84 , srv468791342:84 , srv47341800:84 , srv474166278:84 , srv652518491:84 , srv656945728:84 , srv710544461:84 , srv739336724:84 , srv75293789:84 , srv759864302:84 , srv788528200:84 , srv839036840:84 , srv857949307:84 , srv888467728:84 , srv915702657:84 , srv961387986:84 , srv968929201:84 , srv1005945545:85 , srv1025954206:85 , srv1044207349:85 , srv105525270:85 , srv1090943544:85 , srv1103602659:85 , srv1182746634:85 , srv1250106600:85 , srv1252973397:85 , srv1345749397:85 , srv135133702:85 , srv1434778561:85 , srv1449488821:85 , srv1472286763:85 , srv1486244807:85 , srv156920704:85 , srv1645612041:85 , srv1757445877:85 , srv1779241925:85 , srv1841398453:85 , srv1950961979:85 , srv2044543061:85 , srv2132121390:85 , srv23018263:85 , srv244542276:85 , srv45977721:85 , srv465978652:85 , srv474005805:85 , srv57157002:85 , srv588733703:85 , srv619939092:85 , srv637502859:85 , srv672180431:85 , srv717828340:85 , srv720229892:85 , srv782044236:85 , srv883510:85 , srv887156534:85 , srv905353096:85 , srv963630991:85 , srv422063432:86 , srv454310868:86 , srv928353737:86 , srv1717064651:644 }
2024-03-29 22:53:21,186 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=342 (was 285) - SystemLoadAverage LEAK? -, ProcessCount=170 (was 170), AvailableMemoryMB=2791 (was 3524)
2024-03-29 22:53:21,195 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=342, ProcessCount=170, AvailableMemoryMB=2790
2024-03-29 22:53:21,196 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2024-03-29 22:53:21,197 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc.
2024-03-29 22:53:21,207 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1105104562:84 , srv1182687381:84 , srv1339310234:84 , srv1381755131:84 , srv1452673307:84 , srv1453933463:84 , srv1530492921:84 , srv1602692658:84 , srv1613471534:84 , srv1641164534:84 , srv176304722:84 , srv178080907:84 , srv1793680362:84 , srv1849050545:84 , srv2033664290:84 , srv2040045983:84 , srv2049719007:84 , srv233554201:84 , srv352017286:84 , srv427950997:84 , srv502333013:84 , srv509506766:84 , srv526326685:84 , srv538066657:84 , srv622173083:84 , srv662970496:84 , srv686704966:84 , srv693777114:84 , srv845790924:84 , srv1478156438:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90]
2024-03-29 22:53:21,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-03-29 22:53:21,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv622173083=25, srv2040045983=16, srv1339310234=2, srv2049719007=17, srv1453933463=5, srv1793680362=13, srv233554201=18, srv176304722=11, srv2033664290=15, srv509506766=22, srv1530492921=7, srv662970496=26, srv686704966=27, srv526326685=23, srv1381755131=3, srv1452673307=4, srv1478156438=6, srv538066657=24, srv1613471534=9, srv1602692658=8, srv352017286=19, srv502333013=21, srv178080907=12, srv427950997=20, srv693777114=28, srv845790924=29, srv1641164534=10, srv1849050545=14, srv1105104562=0, srv1182687381=1} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-03-29 22:53:21,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-03-29 22:53:21,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-03-29 22:53:21,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-03-29 22:53:21,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-03-29 22:53:21,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-03-29 22:53:21,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-03-29 22:53:21,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-03-29 22:53:21,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-03-29 22:53:21,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-03-29 22:53:21,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-03-29 22:53:21,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-03-29 22:53:21,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-03-29 22:53:21,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-03-29 22:53:21,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-03-29 22:53:21,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-03-29 22:53:21,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-03-29 22:53:21,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-03-29 22:53:21,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-03-29 22:53:21,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-03-29 22:53:21,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-03-29 22:53:21,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-03-29 22:53:21,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-03-29 22:53:21,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-03-29 22:53:21,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-03-29 22:53:21,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-03-29 22:53:21,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-03-29 22:53:21,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-03-29 22:53:21,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-03-29 22:53:21,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-03-29 22:53:21,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-03-29 22:53:21,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2024-03-29 22:53:21,236 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.165442654731762, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-29 22:53:21,237 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912714347667139, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.165442654731762, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=64800000
2024-03-29 22:55:21,224 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 5377563 different iterations.  Found a solution that moves 1882 regions; Going from a computed imbalance of 0.9912714347667139 to a new imbalance of 4.3926818866725417E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.697037037037037, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-29 22:55:21,230 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1105104562:90 , srv1182687381:90 , srv1339310234:90 , srv1381755131:90 , srv1452673307:90 , srv1453933463:90 , srv1478156438:90 , srv1530492921:90 , srv1602692658:90 , srv1613471534:90 , srv1641164534:90 , srv176304722:90 , srv178080907:90 , srv1793680362:90 , srv1849050545:90 , srv2033664290:90 , srv2040045983:90 , srv2049719007:90 , srv233554201:90 , srv352017286:90 , srv427950997:90 , srv502333013:90 , srv509506766:90 , srv526326685:90 , srv538066657:90 , srv622173083:90 , srv662970496:90 , srv686704966:90 , srv693777114:90 , srv845790924:90 }
2024-03-29 22:55:21,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-03-29 22:55:21,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv622173083=25, srv2040045983=16, srv1339310234=2, srv2049719007=17, srv1453933463=5, srv1793680362=13, srv233554201=18, srv176304722=11, srv2033664290=15, srv509506766=22, srv1530492921=7, srv662970496=26, srv686704966=27, srv526326685=23, srv1381755131=3, srv1452673307=4, srv1478156438=6, srv538066657=24, srv1613471534=9, srv1602692658=8, srv352017286=19, srv502333013=21, srv178080907=12, srv427950997=20, srv693777114=28, srv845790924=29, srv1641164534=10, srv1849050545=14, srv1105104562=0, srv1182687381=1} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-03-29 22:55:21,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-03-29 22:55:21,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-03-29 22:55:21,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-03-29 22:55:21,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-03-29 22:55:21,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-03-29 22:55:21,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-03-29 22:55:21,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-03-29 22:55:21,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-03-29 22:55:21,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-03-29 22:55:21,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-03-29 22:55:21,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-03-29 22:55:21,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-03-29 22:55:21,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-03-29 22:55:21,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-03-29 22:55:21,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-03-29 22:55:21,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-03-29 22:55:21,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-03-29 22:55:21,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-03-29 22:55:21,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-03-29 22:55:21,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-03-29 22:55:21,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-03-29 22:55:21,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-03-29 22:55:21,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-03-29 22:55:21,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-03-29 22:55:21,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-03-29 22:55:21,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-03-29 22:55:21,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-03-29 22:55:21,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-03-29 22:55:21,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-03-29 22:55:21,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-03-29 22:55:21,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2024-03-29 22:55:21,248 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-29 22:55:21,249 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1105104562:90 , srv1182687381:90 , srv1339310234:90 , srv1381755131:90 , srv1452673307:90 , srv1453933463:90 , srv1478156438:90 , srv1530492921:90 , srv1602692658:90 , srv1613471534:90 , srv1641164534:90 , srv176304722:90 , srv178080907:90 , srv1793680362:90 , srv1849050545:90 , srv2033664290:90 , srv2040045983:90 , srv2049719007:90 , srv233554201:90 , srv352017286:90 , srv427950997:90 , srv502333013:90 , srv509506766:90 , srv526326685:90 , srv538066657:90 , srv622173083:90 , srv662970496:90 , srv686704966:90 , srv693777114:90 , srv845790924:90 }
2024-03-29 22:55:21,266 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=206 (was 342), ProcessCount=168 (was 170), AvailableMemoryMB=1973 (was 2790)