2023-05-06 03:53:36,654 INFO  [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins
2023-05-06 03:53:36,993 WARN  [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
2023-05-06 03:53:37,114 INFO  [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2023-05-06 03:53:37,176 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-05-06 03:53:37,184 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-05-06 03:53:37,202 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=338, ProcessCount=179, AvailableMemoryMB=4924
2023-05-06 03:53:37,205 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-05-06 03:53:37,206 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-05-06 03:53:37,343 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1023517460:84 , srv106360746:84 , srv1086914598:84 , srv1088978777:84 , srv1094366683:84 , srv1114599357:84 , srv1116684772:84 , srv1128190827:84 , srv1161923168:84 , srv1169744922:84 , srv1189164011:84 , srv1196195406:84 , srv120656823:84 , srv1220141811:84 , srv1243374569:84 , srv1246230950:84 , srv1253955335:84 , srv126572434:84 , srv1270380784:84 , srv1273567758:84 , srv1280149057:84 , srv1282643310:84 , srv1320116900:84 , srv1395063371:84 , srv1419154157:84 , srv1451200475:84 , srv1452018372:84 , srv1482359352:84 , srv152778155:84 , srv1552137520:84 , srv1560755870:84 , srv1574857613:84 , srv1643255935:84 , srv1657008576:84 , srv1691568845:84 , srv1715153479:84 , srv1721109666:84 , srv1728990370:84 , srv1772222671:84 , srv1774024901:84 , srv1777830310:84 , srv1790869238:84 , srv1800475227:84 , srv1827093145:84 , srv1844483118:84 , srv1878872863:84 , srv1884088825:84 , srv1906517601:84 , srv1979139624:84 , srv1979733523:84 , srv2054556093:84 , srv2071736337:84 , srv2078485376:84 , srv2095002526:84 , srv2125444844:84 , srv227376041:84 , srv249519843:84 , srv257008442:84 , srv273456874:84 , srv293971421:84 , srv3207545:84 , srv328553579:84 , srv392679996:84 , srv39620063:84 , srv433941777:84 , srv438194925:84 , srv449951596:84 , srv457997237:84 , srv488046717:84 , srv49151055:84 , srv499398739:84 , srv540612959:84 , srv558346034:84 , srv57517035:84 , srv577411026:84 , srv582719514:84 , srv608744596:84 , srv647174189:84 , srv651370133:84 , srv652296740:84 , srv656615194:84 , srv659163537:84 , srv660294019:84 , srv661971806:84 , srv663604624:84 , srv736053391:84 , srv741854101:84 , srv753218674:84 , srv771855682:84 , srv781880980:84 , srv782435758:84 , srv812183973:84 , srv849222312:84 , srv869516859:84 , srv951778380:84 , srv955840502:84 , srv973726335:84 , srv981804671:84 , srv99632191:84 , srv2090479042:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90]
2023-05-06 03:53:37,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-05-06 03:53:37,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1451200475=25, srv955840502=96, srv438194925=66, srv3207545=61, srv1560755870=30, srv741854101=87, srv1114599357=5, srv1906517601=47, srv1270380784=18, srv1721109666=36, srv152778155=28, srv647174189=78, srv1715153479=35, srv2078485376=52, srv1884088825=46, srv457997237=68, srv1128190827=7, srv488046717=69, srv1246230950=15, srv1282643310=21, srv257008442=58, srv663604624=85, srv1827093145=43, srv582719514=76, srv106360746=1, srv1161923168=8, srv1280149057=20, srv1552137520=29, srv249519843=57, srv39620063=64, srv49151055=70, srv1273567758=19, srv293971421=60, srv2071736337=51, srv869516859=94, srv782435758=91, srv1878872863=45, srv2095002526=54, srv499398739=71, srv781880980=90, srv558346034=73, srv1574857613=31, srv1189164011=10, srv1790869238=41, srv1086914598=2, srv1777830310=40, srv771855682=89, srv1395063371=23, srv1452018372=26, srv1243374569=14, srv392679996=63, srv120656823=12, srv981804671=98, srv1253955335=16, srv540612959=72, srv651370133=79, srv1657008576=33, srv99632191=99, srv1094366683=4, srv1691568845=34, srv1023517460=0, srv126572434=17, srv449951596=67, srv753218674=88, srv973726335=97, srv577411026=75, srv1116684772=6, srv1320116900=22, srv2090479042=53, srv1169744922=9, srv1728990370=37, srv328553579=62, srv1800475227=42, srv2054556093=50, srv608744596=77, srv1844483118=44, srv1643255935=32, srv951778380=95, srv812183973=92, srv661971806=84, srv1220141811=13, srv1774024901=39, srv433941777=65, srv736053391=86, srv273456874=59, srv227376041=56, srv659163537=82, srv1419154157=24, srv1979733523=49, srv652296740=80, srv2125444844=55, srv1196195406=11, srv1088978777=3, srv1979139624=48, srv660294019=83, srv849222312=93, srv1482359352=27, srv1772222671=38, srv656615194=81, srv57517035=74} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-05-06 03:53:37,452 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-05-06 03:53:37,452 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-05-06 03:53:37,452 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-05-06 03:53:37,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-05-06 03:53:37,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-05-06 03:53:37,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-05-06 03:53:37,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-05-06 03:53:37,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-05-06 03:53:37,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-05-06 03:53:37,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-05-06 03:53:37,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-05-06 03:53:37,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-05-06 03:53:37,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-05-06 03:53:37,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-05-06 03:53:37,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-05-06 03:53:37,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-05-06 03:53:37,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-05-06 03:53:37,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-05-06 03:53:37,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-05-06 03:53:37,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-05-06 03:53:37,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-05-06 03:53:37,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-05-06 03:53:37,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-05-06 03:53:37,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-05-06 03:53:37,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-05-06 03:53:37,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-05-06 03:53:37,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-05-06 03:53:37,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-05-06 03:53:37,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-05-06 03:53:37,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-05-06 03:53:37,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-05-06 03:53:37,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-05-06 03:53:37,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-05-06 03:53:37,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-05-06 03:53:37,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-05-06 03:53:37,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-05-06 03:53:37,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-05-06 03:53:37,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-05-06 03:53:37,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-05-06 03:53:37,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-05-06 03:53:37,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-05-06 03:53:37,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-05-06 03:53:37,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-05-06 03:53:37,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-05-06 03:53:37,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-05-06 03:53:37,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-05-06 03:53:37,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-05-06 03:53:37,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-05-06 03:53:37,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-05-06 03:53:37,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-05-06 03:53:37,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-05-06 03:53:37,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-05-06 03:53:37,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-05-06 03:53:37,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-05-06 03:53:37,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-05-06 03:53:37,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-05-06 03:53:37,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-05-06 03:53:37,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-05-06 03:53:37,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-05-06 03:53:37,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-05-06 03:53:37,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-05-06 03:53:37,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-05-06 03:53:37,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-05-06 03:53:37,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-05-06 03:53:37,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-05-06 03:53:37,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-05-06 03:53:37,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-05-06 03:53:37,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-05-06 03:53:37,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-05-06 03:53:37,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-05-06 03:53:37,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-05-06 03:53:37,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-05-06 03:53:37,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-05-06 03:53:37,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-05-06 03:53:37,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-05-06 03:53:37,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-05-06 03:53:37,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-05-06 03:53:37,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-05-06 03:53:37,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-05-06 03:53:37,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-05-06 03:53:37,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-05-06 03:53:37,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-05-06 03:53:37,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-05-06 03:53:37,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-05-06 03:53:37,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-05-06 03:53:37,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-05-06 03:53:37,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-05-06 03:53:37,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-05-06 03:53:37,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-05-06 03:53:37,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-05-06 03:53:37,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-05-06 03:53:37,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-05-06 03:53:37,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-05-06 03:53:37,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-05-06 03:53:37,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-05-06 03:53:37,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-05-06 03:53:37,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-05-06 03:53:37,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-05-06 03:53:37,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-05-06 03:53:37,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-05-06 03:53:37,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-05-06 03:53:37,869 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-06 03:53:37,869 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-05-06 03:53:37,870 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-05-06 03:53:38,632 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 1132 ms to try 5000 different iterations.  Found a solution that moves 4644 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.1808863654088983. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04062088276641441, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05894127648833263, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.516, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04062088276641441, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17625, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24125, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-06 03:53:38,678 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1128190827:82 , srv1189164011:82 , srv1657008576:82 , srv2071736337:82 , srv2078485376:82 , srv2095002526:82 , srv3207545:82 , srv660294019:82 , srv782435758:82 , srv1094366683:83 , srv1253955335:83 , srv1280149057:83 , srv1772222671:83 , srv753218674:83 , srv1114599357:84 , srv1161923168:84 , srv1320116900:84 , srv1452018372:84 , srv1560755870:84 , srv1721109666:84 , srv1777830310:84 , srv1800475227:84 , srv1979733523:84 , srv457997237:84 , srv741854101:84 , srv812183973:84 , srv973726335:84 , srv1220141811:85 , srv1395063371:85 , srv1552137520:85 , srv1574857613:85 , srv1979139624:85 , srv328553579:85 , srv577411026:85 , srv647174189:85 , srv659163537:85 , srv736053391:85 , srv955840502:85 , srv99632191:85 , srv1419154157:86 , srv1790869238:86 , srv257008442:86 , srv273456874:86 , srv433941777:86 , srv488046717:86 , srv49151055:86 , srv608744596:86 , srv661971806:86 , srv951778380:86 , srv106360746:87 , srv1088978777:87 , srv1116684772:87 , srv1243374569:87 , srv126572434:87 , srv1273567758:87 , srv152778155:87 , srv1774024901:87 , srv1878872863:87 , srv2125444844:87 , srv227376041:87 , srv392679996:87 , srv39620063:87 , srv540612959:87 , srv558346034:87 , srv656615194:87 , srv663604624:87 , srv781880980:87 , srv1023517460:88 , srv1086914598:88 , srv1246230950:88 , srv1643255935:88 , srv1715153479:88 , srv1728990370:88 , srv1827093145:88 , srv1844483118:88 , srv1884088825:88 , srv2054556093:88 , srv771855682:88 , srv849222312:88 , srv869516859:88 , srv1451200475:89 , srv1906517601:89 , srv293971421:89 , srv438194925:89 , srv449951596:89 , srv652296740:89 , srv981804671:89 , srv1196195406:90 , srv120656823:90 , srv1482359352:90 , srv1691568845:90 , srv249519843:90 , srv57517035:90 , srv651370133:90 , srv499398739:91 , srv1270380784:92 , srv1282643310:92 , srv1169744922:93 , srv582719514:93 , srv2090479042:451 }
2023-05-06 03:53:38,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-05-06 03:53:38,685 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1451200475=25, srv955840502=96, srv438194925=66, srv3207545=61, srv1560755870=30, srv741854101=87, srv1114599357=5, srv1906517601=47, srv1270380784=18, srv1721109666=36, srv152778155=28, srv647174189=78, srv1715153479=35, srv2078485376=52, srv1884088825=46, srv457997237=68, srv1128190827=7, srv488046717=69, srv1246230950=15, srv1282643310=21, srv257008442=58, srv663604624=85, srv1827093145=43, srv582719514=76, srv106360746=1, srv1161923168=8, srv1280149057=20, srv1552137520=29, srv249519843=57, srv39620063=64, srv49151055=70, srv1273567758=19, srv293971421=60, srv2071736337=51, srv869516859=94, srv782435758=91, srv1878872863=45, srv2095002526=54, srv499398739=71, srv781880980=90, srv558346034=73, srv1574857613=31, srv1189164011=10, srv1790869238=41, srv1086914598=2, srv1777830310=40, srv771855682=89, srv1395063371=23, srv1452018372=26, srv1243374569=14, srv392679996=63, srv120656823=12, srv981804671=98, srv1253955335=16, srv540612959=72, srv651370133=79, srv1657008576=33, srv99632191=99, srv1094366683=4, srv1691568845=34, srv1023517460=0, srv126572434=17, srv449951596=67, srv753218674=88, srv973726335=97, srv577411026=75, srv1116684772=6, srv1320116900=22, srv2090479042=53, srv1169744922=9, srv1728990370=37, srv328553579=62, srv1800475227=42, srv2054556093=50, srv608744596=77, srv1844483118=44, srv1643255935=32, srv951778380=95, srv812183973=92, srv661971806=84, srv1220141811=13, srv1774024901=39, srv433941777=65, srv736053391=86, srv273456874=59, srv227376041=56, srv659163537=82, srv1419154157=24, srv1979733523=49, srv652296740=80, srv2125444844=55, srv1196195406=11, srv1088978777=3, srv1979139624=48, srv660294019=83, srv849222312=93, srv1482359352=27, srv1772222671=38, srv656615194=81, srv57517035=74} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-05-06 03:53:38,712 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-05-06 03:53:38,712 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-05-06 03:53:38,712 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-05-06 03:53:38,712 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-05-06 03:53:38,712 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-05-06 03:53:38,713 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-05-06 03:53:38,713 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-05-06 03:53:38,713 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-05-06 03:53:38,713 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-05-06 03:53:38,713 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-05-06 03:53:38,713 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-05-06 03:53:38,713 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-05-06 03:53:38,714 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-05-06 03:53:38,714 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-05-06 03:53:38,714 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-05-06 03:53:38,714 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-05-06 03:53:38,714 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-05-06 03:53:38,714 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-05-06 03:53:38,714 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-05-06 03:53:38,714 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-05-06 03:53:38,715 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-05-06 03:53:38,715 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-05-06 03:53:38,715 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-05-06 03:53:38,715 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-05-06 03:53:38,715 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-05-06 03:53:38,715 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-05-06 03:53:38,715 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-05-06 03:53:38,715 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-05-06 03:53:38,715 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-05-06 03:53:38,716 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-05-06 03:53:38,716 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-05-06 03:53:38,716 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-05-06 03:53:38,716 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-05-06 03:53:38,716 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-05-06 03:53:38,716 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-05-06 03:53:38,716 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-05-06 03:53:38,716 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-05-06 03:53:38,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-05-06 03:53:38,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-05-06 03:53:38,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-05-06 03:53:38,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-05-06 03:53:38,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-05-06 03:53:38,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-05-06 03:53:38,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-05-06 03:53:38,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-05-06 03:53:38,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-05-06 03:53:38,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-05-06 03:53:38,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-05-06 03:53:38,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-05-06 03:53:38,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-05-06 03:53:38,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-05-06 03:53:38,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-05-06 03:53:38,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-05-06 03:53:38,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-05-06 03:53:38,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-05-06 03:53:38,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-05-06 03:53:38,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-05-06 03:53:38,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-05-06 03:53:38,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-05-06 03:53:38,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-05-06 03:53:38,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-05-06 03:53:38,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-05-06 03:53:38,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-05-06 03:53:38,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-05-06 03:53:38,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-05-06 03:53:38,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-05-06 03:53:38,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-05-06 03:53:38,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-05-06 03:53:38,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-05-06 03:53:38,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-05-06 03:53:38,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-05-06 03:53:38,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-05-06 03:53:38,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-05-06 03:53:38,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-05-06 03:53:38,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-05-06 03:53:38,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-05-06 03:53:38,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-05-06 03:53:38,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-05-06 03:53:38,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-05-06 03:53:38,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-05-06 03:53:38,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-05-06 03:53:38,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-05-06 03:53:38,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-05-06 03:53:38,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-05-06 03:53:38,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-05-06 03:53:38,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-05-06 03:53:38,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-05-06 03:53:38,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-05-06 03:53:38,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-05-06 03:53:38,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-05-06 03:53:38,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-05-06 03:53:38,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-05-06 03:53:38,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-05-06 03:53:38,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-05-06 03:53:38,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-05-06 03:53:38,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-05-06 03:53:38,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-05-06 03:53:38,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-05-06 03:53:38,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-05-06 03:53:38,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-05-06 03:53:38,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-05-06 03:53:38,895 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04062088276641441, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05894127648833263, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04062088276641441, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17625, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24125, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-06 03:53:38,895 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-05-06 03:53:38,896 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.18085384742587754, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04062088276641441, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05894127648833263, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04062088276641441, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17625, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24125, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-05-06 03:53:39,196 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 441 ms to try 5000 different iterations.  Found a solution that moves 2643 regions; Going from a computed imbalance of 0.18085384742587754 to a new imbalance of 0.015890281238763154. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.019938176049918176); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.029644356609443877, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.2936666666666667, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.019938176049918176); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.013083333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.042916666666666665, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-06 03:53:39,204 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv582719514:80 , srv1169744922:81 , srv1728990370:81 , srv293971421:81 , srv499398739:81 , srv1023517460:82 , srv1270380784:82 , srv1282643310:82 , srv1844483118:82 , srv249519843:82 , srv449951596:82 , srv652296740:82 , srv849222312:82 , srv106360746:83 , srv1116684772:83 , srv120656823:83 , srv1246230950:83 , srv1482359352:83 , srv1691568845:83 , srv1715153479:83 , srv438194925:83 , srv540612959:83 , srv558346034:83 , srv57517035:83 , srv651370133:83 , srv869516859:83 , srv1086914598:84 , srv1088978777:84 , srv1419154157:84 , srv1451200475:84 , srv1643255935:84 , srv1774024901:84 , srv1884088825:84 , srv392679996:84 , srv39620063:84 , srv781880980:84 , srv126572434:85 , srv152778155:85 , srv1574857613:85 , srv273456874:85 , srv49151055:85 , srv659163537:85 , srv663604624:85 , srv771855682:85 , srv981804671:85 , srv1196195406:86 , srv1220141811:86 , srv1552137520:86 , srv1790869238:86 , srv1878872863:86 , srv1906517601:86 , srv2054556093:86 , srv2125444844:86 , srv227376041:86 , srv257008442:86 , srv608744596:86 , srv656615194:86 , srv661971806:86 , srv736053391:86 , srv1243374569:87 , srv1395063371:87 , srv1827093145:87 , srv1979733523:87 , srv457997237:87 , srv488046717:87 , srv577411026:87 , srv647174189:87 , srv973726335:87 , srv1094366683:88 , srv1161923168:88 , srv1320116900:88 , srv1560755870:88 , srv1657008576:88 , srv1721109666:88 , srv1772222671:88 , srv812183973:88 , srv951778380:88 , srv1114599357:89 , srv1128190827:89 , srv1253955335:89 , srv1800475227:89 , srv328553579:89 , srv433941777:89 , srv741854101:89 , srv955840502:89 , srv99632191:89 , srv1189164011:90 , srv1273567758:90 , srv1452018372:90 , srv1777830310:90 , srv1979139624:90 , srv2071736337:90 , srv2078485376:90 , srv3207545:90 , srv660294019:90 , srv753218674:90 , srv1280149057:91 , srv2095002526:92 , srv782435758:93 , srv2090479042:500 }
2023-05-06 03:53:39,207 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-05-06 03:53:39,208 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1451200475=25, srv955840502=96, srv438194925=66, srv3207545=61, srv1560755870=30, srv741854101=87, srv1114599357=5, srv1906517601=47, srv1270380784=18, srv1721109666=36, srv152778155=28, srv647174189=78, srv1715153479=35, srv2078485376=52, srv1884088825=46, srv457997237=68, srv1128190827=7, srv488046717=69, srv1246230950=15, srv1282643310=21, srv257008442=58, srv663604624=85, srv1827093145=43, srv582719514=76, srv106360746=1, srv1161923168=8, srv1280149057=20, srv1552137520=29, srv249519843=57, srv39620063=64, srv49151055=70, srv1273567758=19, srv293971421=60, srv2071736337=51, srv869516859=94, srv782435758=91, srv1878872863=45, srv2095002526=54, srv499398739=71, srv781880980=90, srv558346034=73, srv1574857613=31, srv1189164011=10, srv1790869238=41, srv1086914598=2, srv1777830310=40, srv771855682=89, srv1395063371=23, srv1452018372=26, srv1243374569=14, srv392679996=63, srv120656823=12, srv981804671=98, srv1253955335=16, srv540612959=72, srv651370133=79, srv1657008576=33, srv99632191=99, srv1094366683=4, srv1691568845=34, srv1023517460=0, srv126572434=17, srv449951596=67, srv753218674=88, srv973726335=97, srv577411026=75, srv1116684772=6, srv1320116900=22, srv2090479042=53, srv1169744922=9, srv1728990370=37, srv328553579=62, srv1800475227=42, srv2054556093=50, srv608744596=77, srv1844483118=44, srv1643255935=32, srv951778380=95, srv812183973=92, srv661971806=84, srv1220141811=13, srv1774024901=39, srv433941777=65, srv736053391=86, srv273456874=59, srv227376041=56, srv659163537=82, srv1419154157=24, srv1979733523=49, srv652296740=80, srv2125444844=55, srv1196195406=11, srv1088978777=3, srv1979139624=48, srv660294019=83, srv849222312=93, srv1482359352=27, srv1772222671=38, srv656615194=81, srv57517035=74} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-05-06 03:53:39,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-05-06 03:53:39,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-05-06 03:53:39,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-05-06 03:53:39,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-05-06 03:53:39,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-05-06 03:53:39,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-05-06 03:53:39,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-05-06 03:53:39,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-05-06 03:53:39,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-05-06 03:53:39,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-05-06 03:53:39,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-05-06 03:53:39,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-05-06 03:53:39,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-05-06 03:53:39,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-05-06 03:53:39,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-05-06 03:53:39,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-05-06 03:53:39,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-05-06 03:53:39,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-05-06 03:53:39,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-05-06 03:53:39,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-05-06 03:53:39,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-05-06 03:53:39,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-05-06 03:53:39,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-05-06 03:53:39,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-05-06 03:53:39,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-05-06 03:53:39,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-05-06 03:53:39,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-05-06 03:53:39,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-05-06 03:53:39,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-05-06 03:53:39,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-05-06 03:53:39,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-05-06 03:53:39,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-05-06 03:53:39,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-05-06 03:53:39,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-05-06 03:53:39,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-05-06 03:53:39,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-05-06 03:53:39,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-05-06 03:53:39,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-05-06 03:53:39,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-05-06 03:53:39,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-05-06 03:53:39,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-05-06 03:53:39,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-05-06 03:53:39,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-05-06 03:53:39,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-05-06 03:53:39,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-05-06 03:53:39,228 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-05-06 03:53:39,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-05-06 03:53:39,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-05-06 03:53:39,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-05-06 03:53:39,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-05-06 03:53:39,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-05-06 03:53:39,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-05-06 03:53:39,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-05-06 03:53:39,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-05-06 03:53:39,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-05-06 03:53:39,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-05-06 03:53:39,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-05-06 03:53:39,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-05-06 03:53:39,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-05-06 03:53:39,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-05-06 03:53:39,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-05-06 03:53:39,229 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-05-06 03:53:39,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-05-06 03:53:39,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-05-06 03:53:39,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-05-06 03:53:39,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-05-06 03:53:39,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-05-06 03:53:39,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-05-06 03:53:39,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-05-06 03:53:39,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-05-06 03:53:39,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-05-06 03:53:39,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-05-06 03:53:39,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-05-06 03:53:39,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-05-06 03:53:39,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-05-06 03:53:39,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-05-06 03:53:39,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-05-06 03:53:39,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-05-06 03:53:39,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-05-06 03:53:39,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-05-06 03:53:39,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-05-06 03:53:39,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-05-06 03:53:39,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-05-06 03:53:39,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-05-06 03:53:39,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-05-06 03:53:39,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-05-06 03:53:39,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-05-06 03:53:39,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-05-06 03:53:39,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-05-06 03:53:39,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-05-06 03:53:39,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-05-06 03:53:39,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-05-06 03:53:39,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-05-06 03:53:39,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-05-06 03:53:39,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-05-06 03:53:39,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-05-06 03:53:39,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-05-06 03:53:39,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-05-06 03:53:39,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-05-06 03:53:39,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-05-06 03:53:39,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-05-06 03:53:39,366 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.019938176049918176); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.029644356609443877, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.019938176049918176); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.013083333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.042916666666666665, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-06 03:53:39,366 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-05-06 03:53:39,367 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.015871774557211917, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.019938176049918176); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.029644356609443877, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.019938176049918176); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.013083333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.042916666666666665, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-05-06 03:53:39,675 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 413 ms to try 5000 different iterations.  Found a solution that moves 1035 regions; Going from a computed imbalance of 0.015871774557211917 to a new imbalance of 3.61771881220213E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00610013298189536); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00723192820996717); MoveCostFunction : (multiplier=7.0, imbalance=0.115, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.00610013298189536); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00325); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-06 03:53:39,677 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1196195406:81 , srv782435758:81 , srv1273567758:82 , srv1906517601:82 , srv1243374569:83 , srv1777830310:83 , srv1827093145:83 , srv1979139624:83 , srv433941777:83 , srv1114599357:84 , srv1169744922:84 , srv120656823:84 , srv1270380784:84 , srv1280149057:84 , srv1282643310:84 , srv1452018372:84 , srv1482359352:84 , srv1878872863:84 , srv2054556093:84 , srv2071736337:84 , srv2095002526:84 , srv2125444844:84 , srv227376041:84 , srv328553579:84 , srv488046717:84 , srv753218674:84 , srv951778380:84 , srv955840502:84 , srv981804671:84 , srv1086914598:85 , srv1161923168:85 , srv1253955335:85 , srv126572434:85 , srv1320116900:85 , srv1451200475:85 , srv152778155:85 , srv1560755870:85 , srv1643255935:85 , srv1691568845:85 , srv1721109666:85 , srv1772222671:85 , srv1800475227:85 , srv1884088825:85 , srv249519843:85 , srv438194925:85 , srv57517035:85 , srv577411026:85 , srv647174189:85 , srv651370133:85 , srv656615194:85 , srv660294019:85 , srv661971806:85 , srv663604624:85 , srv741854101:85 , srv771855682:85 , srv99632191:85 , srv1088978777:86 , srv1094366683:86 , srv1128190827:86 , srv1189164011:86 , srv1220141811:86 , srv1246230950:86 , srv1395063371:86 , srv1552137520:86 , srv1790869238:86 , srv1979733523:86 , srv2078485376:86 , srv257008442:86 , srv273456874:86 , srv3207545:86 , srv392679996:86 , srv39620063:86 , srv457997237:86 , srv49151055:86 , srv499398739:86 , srv582719514:86 , srv608744596:86 , srv652296740:86 , srv736053391:86 , srv781880980:86 , srv812183973:86 , srv1023517460:87 , srv1419154157:87 , srv1574857613:87 , srv1657008576:87 , srv1715153479:87 , srv1774024901:87 , srv293971421:87 , srv449951596:87 , srv540612959:87 , srv659163537:87 , srv849222312:87 , srv869516859:87 , srv973726335:87 , srv106360746:88 , srv1844483118:88 , srv558346034:88 , srv1116684772:89 , srv1728990370:89 , srv2090479042:561 }
2023-05-06 03:53:39,683 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-05-06 03:53:39,684 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1451200475=25, srv955840502=96, srv438194925=66, srv3207545=61, srv1560755870=30, srv741854101=87, srv1114599357=5, srv1906517601=47, srv1270380784=18, srv1721109666=36, srv152778155=28, srv647174189=78, srv1715153479=35, srv2078485376=52, srv1884088825=46, srv457997237=68, srv1128190827=7, srv488046717=69, srv1246230950=15, srv1282643310=21, srv257008442=58, srv663604624=85, srv1827093145=43, srv582719514=76, srv106360746=1, srv1161923168=8, srv1280149057=20, srv1552137520=29, srv249519843=57, srv39620063=64, srv49151055=70, srv1273567758=19, srv293971421=60, srv2071736337=51, srv869516859=94, srv782435758=91, srv1878872863=45, srv2095002526=54, srv499398739=71, srv781880980=90, srv558346034=73, srv1574857613=31, srv1189164011=10, srv1790869238=41, srv1086914598=2, srv1777830310=40, srv771855682=89, srv1395063371=23, srv1452018372=26, srv1243374569=14, srv392679996=63, srv120656823=12, srv981804671=98, srv1253955335=16, srv540612959=72, srv651370133=79, srv1657008576=33, srv99632191=99, srv1094366683=4, srv1691568845=34, srv1023517460=0, srv126572434=17, srv449951596=67, srv753218674=88, srv973726335=97, srv577411026=75, srv1116684772=6, srv1320116900=22, srv2090479042=53, srv1169744922=9, srv1728990370=37, srv328553579=62, srv1800475227=42, srv2054556093=50, srv608744596=77, srv1844483118=44, srv1643255935=32, srv951778380=95, srv812183973=92, srv661971806=84, srv1220141811=13, srv1774024901=39, srv433941777=65, srv736053391=86, srv273456874=59, srv227376041=56, srv659163537=82, srv1419154157=24, srv1979733523=49, srv652296740=80, srv2125444844=55, srv1196195406=11, srv1088978777=3, srv1979139624=48, srv660294019=83, srv849222312=93, srv1482359352=27, srv1772222671=38, srv656615194=81, srv57517035=74} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-05-06 03:53:39,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-05-06 03:53:39,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-05-06 03:53:39,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-05-06 03:53:39,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-05-06 03:53:39,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-05-06 03:53:39,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-05-06 03:53:39,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-05-06 03:53:39,701 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-05-06 03:53:39,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-05-06 03:53:39,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-05-06 03:53:39,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-05-06 03:53:39,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-05-06 03:53:39,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-05-06 03:53:39,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-05-06 03:53:39,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-05-06 03:53:39,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-05-06 03:53:39,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-05-06 03:53:39,702 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-05-06 03:53:39,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-05-06 03:53:39,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-05-06 03:53:39,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-05-06 03:53:39,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-05-06 03:53:39,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-05-06 03:53:39,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-05-06 03:53:39,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-05-06 03:53:39,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-05-06 03:53:39,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-05-06 03:53:39,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-05-06 03:53:39,703 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-05-06 03:53:39,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-05-06 03:53:39,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-05-06 03:53:39,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-05-06 03:53:39,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-05-06 03:53:39,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-05-06 03:53:39,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-05-06 03:53:39,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-05-06 03:53:39,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-05-06 03:53:39,704 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-05-06 03:53:39,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-05-06 03:53:39,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-05-06 03:53:39,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-05-06 03:53:39,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-05-06 03:53:39,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-05-06 03:53:39,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-05-06 03:53:39,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-05-06 03:53:39,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-05-06 03:53:39,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-05-06 03:53:39,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-05-06 03:53:39,705 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-05-06 03:53:39,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-05-06 03:53:39,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-05-06 03:53:39,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-05-06 03:53:39,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-05-06 03:53:39,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-05-06 03:53:39,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-05-06 03:53:39,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-05-06 03:53:39,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-05-06 03:53:39,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-05-06 03:53:39,706 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-05-06 03:53:39,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-05-06 03:53:39,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-05-06 03:53:39,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-05-06 03:53:39,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-05-06 03:53:39,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-05-06 03:53:39,707 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-05-06 03:53:39,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-05-06 03:53:39,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-05-06 03:53:39,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-05-06 03:53:39,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-05-06 03:53:39,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-05-06 03:53:39,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-05-06 03:53:39,708 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-05-06 03:53:39,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-05-06 03:53:39,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-05-06 03:53:39,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-05-06 03:53:39,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-05-06 03:53:39,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-05-06 03:53:39,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-05-06 03:53:39,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-05-06 03:53:39,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-05-06 03:53:39,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-05-06 03:53:39,709 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-05-06 03:53:39,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-05-06 03:53:39,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-05-06 03:53:39,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-05-06 03:53:39,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-05-06 03:53:39,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-05-06 03:53:39,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-05-06 03:53:39,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-05-06 03:53:39,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-05-06 03:53:39,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-05-06 03:53:39,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-05-06 03:53:39,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-05-06 03:53:39,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-05-06 03:53:39,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-05-06 03:53:39,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-05-06 03:53:39,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-05-06 03:53:39,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-05-06 03:53:39,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-05-06 03:53:39,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-05-06 03:53:39,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-05-06 03:53:39,874 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00610013298189536); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00723192820996717); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.00610013298189536); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00325); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-06 03:53:39,874 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-05-06 03:53:39,874 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=3.545246563221693E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00610013298189536); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00723192820996717); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.00610013298189536); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00325); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-05-06 03:53:40,308 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 576 ms to try 5000 different iterations.  Found a solution that moves 257 regions; Going from a computed imbalance of 3.545246563221693E-4 to a new imbalance of 3.5759964822652355E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.994073055283275E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.002010075630518424); MoveCostFunction : (multiplier=7.0, imbalance=0.028555555555555556, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.994073055283275E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=2.5E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-06 03:53:40,309 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1116684772:83 , srv1273567758:83 , srv1728990370:83 , srv582719514:83 , srv106360746:84 , srv1169744922:84 , srv1189164011:84 , srv120656823:84 , srv1220141811:84 , srv1280149057:84 , srv1282643310:84 , srv1320116900:84 , srv1395063371:84 , srv1451200475:84 , srv1452018372:84 , srv1574857613:84 , srv1691568845:84 , srv1715153479:84 , srv1774024901:84 , srv1790869238:84 , srv1800475227:84 , srv1827093145:84 , srv1844483118:84 , srv1884088825:84 , srv1979139624:84 , srv2054556093:84 , srv2095002526:84 , srv257008442:84 , srv293971421:84 , srv3207545:84 , srv328553579:84 , srv433941777:84 , srv499398739:84 , srv558346034:84 , srv608744596:84 , srv651370133:84 , srv656615194:84 , srv663604624:84 , srv736053391:84 , srv741854101:84 , srv771855682:84 , srv812183973:84 , srv849222312:84 , srv869516859:84 , srv955840502:84 , srv973726335:84 , srv981804671:84 , srv99632191:84 , srv1023517460:85 , srv1086914598:85 , srv1088978777:85 , srv1094366683:85 , srv1114599357:85 , srv1128190827:85 , srv1161923168:85 , srv1196195406:85 , srv1246230950:85 , srv1253955335:85 , srv126572434:85 , srv1270380784:85 , srv1419154157:85 , srv1482359352:85 , srv152778155:85 , srv1552137520:85 , srv1560755870:85 , srv1643255935:85 , srv1721109666:85 , srv1777830310:85 , srv1878872863:85 , srv1979733523:85 , srv2078485376:85 , srv2125444844:85 , srv227376041:85 , srv249519843:85 , srv273456874:85 , srv392679996:85 , srv39620063:85 , srv438194925:85 , srv449951596:85 , srv457997237:85 , srv488046717:85 , srv49151055:85 , srv540612959:85 , srv57517035:85 , srv577411026:85 , srv647174189:85 , srv652296740:85 , srv659163537:85 , srv660294019:85 , srv661971806:85 , srv753218674:85 , srv781880980:85 , srv951778380:85 , srv1243374569:86 , srv1657008576:86 , srv1772222671:86 , srv1906517601:86 , srv2071736337:86 , srv782435758:86 , srv2090479042:631 }
2023-05-06 03:53:40,315 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-05-06 03:53:40,316 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1451200475=25, srv955840502=96, srv438194925=66, srv3207545=61, srv1560755870=30, srv741854101=87, srv1114599357=5, srv1906517601=47, srv1270380784=18, srv1721109666=36, srv152778155=28, srv647174189=78, srv1715153479=35, srv2078485376=52, srv1884088825=46, srv457997237=68, srv1128190827=7, srv488046717=69, srv1246230950=15, srv1282643310=21, srv257008442=58, srv663604624=85, srv1827093145=43, srv582719514=76, srv106360746=1, srv1161923168=8, srv1280149057=20, srv1552137520=29, srv249519843=57, srv39620063=64, srv49151055=70, srv1273567758=19, srv293971421=60, srv2071736337=51, srv869516859=94, srv782435758=91, srv1878872863=45, srv2095002526=54, srv499398739=71, srv781880980=90, srv558346034=73, srv1574857613=31, srv1189164011=10, srv1790869238=41, srv1086914598=2, srv1777830310=40, srv771855682=89, srv1395063371=23, srv1452018372=26, srv1243374569=14, srv392679996=63, srv120656823=12, srv981804671=98, srv1253955335=16, srv540612959=72, srv651370133=79, srv1657008576=33, srv99632191=99, srv1094366683=4, srv1691568845=34, srv1023517460=0, srv126572434=17, srv449951596=67, srv753218674=88, srv973726335=97, srv577411026=75, srv1116684772=6, srv1320116900=22, srv2090479042=53, srv1169744922=9, srv1728990370=37, srv328553579=62, srv1800475227=42, srv2054556093=50, srv608744596=77, srv1844483118=44, srv1643255935=32, srv951778380=95, srv812183973=92, srv661971806=84, srv1220141811=13, srv1774024901=39, srv433941777=65, srv736053391=86, srv273456874=59, srv227376041=56, srv659163537=82, srv1419154157=24, srv1979733523=49, srv652296740=80, srv2125444844=55, srv1196195406=11, srv1088978777=3, srv1979139624=48, srv660294019=83, srv849222312=93, srv1482359352=27, srv1772222671=38, srv656615194=81, srv57517035=74} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-05-06 03:53:40,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-05-06 03:53:40,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-05-06 03:53:40,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-05-06 03:53:40,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-05-06 03:53:40,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-05-06 03:53:40,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-05-06 03:53:40,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-05-06 03:53:40,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-05-06 03:53:40,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-05-06 03:53:40,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-05-06 03:53:40,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-05-06 03:53:40,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-05-06 03:53:40,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-05-06 03:53:40,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-05-06 03:53:40,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-05-06 03:53:40,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-05-06 03:53:40,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-05-06 03:53:40,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-05-06 03:53:40,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-05-06 03:53:40,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-05-06 03:53:40,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-05-06 03:53:40,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-05-06 03:53:40,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-05-06 03:53:40,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-05-06 03:53:40,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-05-06 03:53:40,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-05-06 03:53:40,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-05-06 03:53:40,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-05-06 03:53:40,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-05-06 03:53:40,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-05-06 03:53:40,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-05-06 03:53:40,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-05-06 03:53:40,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-05-06 03:53:40,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-05-06 03:53:40,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-05-06 03:53:40,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-05-06 03:53:40,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-05-06 03:53:40,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-05-06 03:53:40,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-05-06 03:53:40,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-05-06 03:53:40,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-05-06 03:53:40,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-05-06 03:53:40,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-05-06 03:53:40,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-05-06 03:53:40,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-05-06 03:53:40,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-05-06 03:53:40,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-05-06 03:53:40,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-05-06 03:53:40,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-05-06 03:53:40,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-05-06 03:53:40,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-05-06 03:53:40,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-05-06 03:53:40,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-05-06 03:53:40,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-05-06 03:53:40,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-05-06 03:53:40,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-05-06 03:53:40,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-05-06 03:53:40,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-05-06 03:53:40,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-05-06 03:53:40,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-05-06 03:53:40,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-05-06 03:53:40,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-05-06 03:53:40,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-05-06 03:53:40,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-05-06 03:53:40,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-05-06 03:53:40,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-05-06 03:53:40,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-05-06 03:53:40,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-05-06 03:53:40,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-05-06 03:53:40,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-05-06 03:53:40,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-05-06 03:53:40,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-05-06 03:53:40,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-05-06 03:53:40,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-05-06 03:53:40,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-05-06 03:53:40,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-05-06 03:53:40,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-05-06 03:53:40,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-05-06 03:53:40,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-05-06 03:53:40,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-05-06 03:53:40,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-05-06 03:53:40,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-05-06 03:53:40,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-05-06 03:53:40,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-05-06 03:53:40,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-05-06 03:53:40,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-05-06 03:53:40,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-05-06 03:53:40,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-05-06 03:53:40,339 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-05-06 03:53:40,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-05-06 03:53:40,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-05-06 03:53:40,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-05-06 03:53:40,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-05-06 03:53:40,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-05-06 03:53:40,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-05-06 03:53:40,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-05-06 03:53:40,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-05-06 03:53:40,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-05-06 03:53:40,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-05-06 03:53:40,340 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-05-06 03:53:40,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-05-06 03:53:40,485 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.994073055283275E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.002010075630518424); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.994073055283275E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=2.5E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-06 03:53:40,485 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-05-06 03:53:40,485 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=3.396041236004634E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.994073055283275E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.002010075630518424); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.994073055283275E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=2.5E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-05-06 03:53:40,772 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 413 ms to try 5000 different iterations.  Found a solution that moves 38 regions; Going from a computed imbalance of 3.396041236004634E-5 to a new imbalance of 6.298160426806302E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0013400504203456161); MoveCostFunction : (multiplier=7.0, imbalance=0.004222222222222222); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-06 03:53:40,772 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1243374569:83 , srv1270380784:83 , srv1657008576:83 , srv1906517601:83 , srv2078485376:83 , srv449951596:83 , srv57517035:83 , srv951778380:83 , srv99632191:83 , srv1023517460:84 , srv106360746:84 , srv1086914598:84 , srv1088978777:84 , srv1094366683:84 , srv1114599357:84 , srv1116684772:84 , srv1128190827:84 , srv1161923168:84 , srv1169744922:84 , srv1189164011:84 , srv1196195406:84 , srv1246230950:84 , srv1253955335:84 , srv126572434:84 , srv1273567758:84 , srv1280149057:84 , srv1282643310:84 , srv1395063371:84 , srv1419154157:84 , srv1451200475:84 , srv1452018372:84 , srv1482359352:84 , srv152778155:84 , srv1552137520:84 , srv1560755870:84 , srv1643255935:84 , srv1691568845:84 , srv1715153479:84 , srv1721109666:84 , srv1772222671:84 , srv1774024901:84 , srv1777830310:84 , srv1790869238:84 , srv1800475227:84 , srv1827093145:84 , srv1844483118:84 , srv1878872863:84 , srv1979139624:84 , srv1979733523:84 , srv2054556093:84 , srv2071736337:84 , srv2095002526:84 , srv2125444844:84 , srv227376041:84 , srv249519843:84 , srv257008442:84 , srv273456874:84 , srv3207545:84 , srv328553579:84 , srv392679996:84 , srv39620063:84 , srv433941777:84 , srv438194925:84 , srv457997237:84 , srv488046717:84 , srv49151055:84 , srv499398739:84 , srv540612959:84 , srv558346034:84 , srv577411026:84 , srv582719514:84 , srv608744596:84 , srv647174189:84 , srv651370133:84 , srv652296740:84 , srv656615194:84 , srv659163537:84 , srv660294019:84 , srv661971806:84 , srv741854101:84 , srv753218674:84 , srv771855682:84 , srv781880980:84 , srv782435758:84 , srv812183973:84 , srv869516859:84 , srv955840502:84 , srv973726335:84 , srv981804671:84 , srv120656823:85 , srv1220141811:85 , srv1320116900:85 , srv1574857613:85 , srv1728990370:85 , srv1884088825:85 , srv293971421:85 , srv663604624:85 , srv736053391:85 , srv849222312:85 , srv2090479042:683 }
2023-05-06 03:53:40,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-05-06 03:53:40,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1451200475=25, srv955840502=96, srv438194925=66, srv3207545=61, srv1560755870=30, srv741854101=87, srv1114599357=5, srv1906517601=47, srv1270380784=18, srv1721109666=36, srv152778155=28, srv647174189=78, srv1715153479=35, srv2078485376=52, srv1884088825=46, srv457997237=68, srv1128190827=7, srv488046717=69, srv1246230950=15, srv1282643310=21, srv257008442=58, srv663604624=85, srv1827093145=43, srv582719514=76, srv106360746=1, srv1161923168=8, srv1280149057=20, srv1552137520=29, srv249519843=57, srv39620063=64, srv49151055=70, srv1273567758=19, srv293971421=60, srv2071736337=51, srv869516859=94, srv782435758=91, srv1878872863=45, srv2095002526=54, srv499398739=71, srv781880980=90, srv558346034=73, srv1574857613=31, srv1189164011=10, srv1790869238=41, srv1086914598=2, srv1777830310=40, srv771855682=89, srv1395063371=23, srv1452018372=26, srv1243374569=14, srv392679996=63, srv120656823=12, srv981804671=98, srv1253955335=16, srv540612959=72, srv651370133=79, srv1657008576=33, srv99632191=99, srv1094366683=4, srv1691568845=34, srv1023517460=0, srv126572434=17, srv449951596=67, srv753218674=88, srv973726335=97, srv577411026=75, srv1116684772=6, srv1320116900=22, srv2090479042=53, srv1169744922=9, srv1728990370=37, srv328553579=62, srv1800475227=42, srv2054556093=50, srv608744596=77, srv1844483118=44, srv1643255935=32, srv951778380=95, srv812183973=92, srv661971806=84, srv1220141811=13, srv1774024901=39, srv433941777=65, srv736053391=86, srv273456874=59, srv227376041=56, srv659163537=82, srv1419154157=24, srv1979733523=49, srv652296740=80, srv2125444844=55, srv1196195406=11, srv1088978777=3, srv1979139624=48, srv660294019=83, srv849222312=93, srv1482359352=27, srv1772222671=38, srv656615194=81, srv57517035=74} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-05-06 03:53:40,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-05-06 03:53:40,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-05-06 03:53:40,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-05-06 03:53:40,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-05-06 03:53:40,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-05-06 03:53:40,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-05-06 03:53:40,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-05-06 03:53:40,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-05-06 03:53:40,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-05-06 03:53:40,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-05-06 03:53:40,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-05-06 03:53:40,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-05-06 03:53:40,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-05-06 03:53:40,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-05-06 03:53:40,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-05-06 03:53:40,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-05-06 03:53:40,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-05-06 03:53:40,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-05-06 03:53:40,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-05-06 03:53:40,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-05-06 03:53:40,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-05-06 03:53:40,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-05-06 03:53:40,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-05-06 03:53:40,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-05-06 03:53:40,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-05-06 03:53:40,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-05-06 03:53:40,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-05-06 03:53:40,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-05-06 03:53:40,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-05-06 03:53:40,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-05-06 03:53:40,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-05-06 03:53:40,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-05-06 03:53:40,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-05-06 03:53:40,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-05-06 03:53:40,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-05-06 03:53:40,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-05-06 03:53:40,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-05-06 03:53:40,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-05-06 03:53:40,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-05-06 03:53:40,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-05-06 03:53:40,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-05-06 03:53:40,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-05-06 03:53:40,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-05-06 03:53:40,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-05-06 03:53:40,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-05-06 03:53:40,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-05-06 03:53:40,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-05-06 03:53:40,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-05-06 03:53:40,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-05-06 03:53:40,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-05-06 03:53:40,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-05-06 03:53:40,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-05-06 03:53:40,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-05-06 03:53:40,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-05-06 03:53:40,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-05-06 03:53:40,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-05-06 03:53:40,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-05-06 03:53:40,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-05-06 03:53:40,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-05-06 03:53:40,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-05-06 03:53:40,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-05-06 03:53:40,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-05-06 03:53:40,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-05-06 03:53:40,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-05-06 03:53:40,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-05-06 03:53:40,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-05-06 03:53:40,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-05-06 03:53:40,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-05-06 03:53:40,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-05-06 03:53:40,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-05-06 03:53:40,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-05-06 03:53:40,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-05-06 03:53:40,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-05-06 03:53:40,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-05-06 03:53:40,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-05-06 03:53:40,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-05-06 03:53:40,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-05-06 03:53:40,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-05-06 03:53:40,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-05-06 03:53:40,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-05-06 03:53:40,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-05-06 03:53:40,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-05-06 03:53:40,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-05-06 03:53:40,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-05-06 03:53:40,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-05-06 03:53:40,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-05-06 03:53:40,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-05-06 03:53:40,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-05-06 03:53:40,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-05-06 03:53:40,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-05-06 03:53:40,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-05-06 03:53:40,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-05-06 03:53:40,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-05-06 03:53:40,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-05-06 03:53:40,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-05-06 03:53:40,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-05-06 03:53:40,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-05-06 03:53:40,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-05-06 03:53:40,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-05-06 03:53:40,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-05-06 03:53:40,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-05-06 03:53:40,918 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=6.032078739728369E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0013400504203456161); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-06 03:53:40,918 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1243374569:83 , srv1270380784:83 , srv1657008576:83 , srv1906517601:83 , srv2078485376:83 , srv449951596:83 , srv57517035:83 , srv951778380:83 , srv99632191:83 , srv1023517460:84 , srv106360746:84 , srv1086914598:84 , srv1088978777:84 , srv1094366683:84 , srv1114599357:84 , srv1116684772:84 , srv1128190827:84 , srv1161923168:84 , srv1169744922:84 , srv1189164011:84 , srv1196195406:84 , srv1246230950:84 , srv1253955335:84 , srv126572434:84 , srv1273567758:84 , srv1280149057:84 , srv1282643310:84 , srv1395063371:84 , srv1419154157:84 , srv1451200475:84 , srv1452018372:84 , srv1482359352:84 , srv152778155:84 , srv1552137520:84 , srv1560755870:84 , srv1643255935:84 , srv1691568845:84 , srv1715153479:84 , srv1721109666:84 , srv1772222671:84 , srv1774024901:84 , srv1777830310:84 , srv1790869238:84 , srv1800475227:84 , srv1827093145:84 , srv1844483118:84 , srv1878872863:84 , srv1979139624:84 , srv1979733523:84 , srv2054556093:84 , srv2071736337:84 , srv2095002526:84 , srv2125444844:84 , srv227376041:84 , srv249519843:84 , srv257008442:84 , srv273456874:84 , srv3207545:84 , srv328553579:84 , srv392679996:84 , srv39620063:84 , srv433941777:84 , srv438194925:84 , srv457997237:84 , srv488046717:84 , srv49151055:84 , srv499398739:84 , srv540612959:84 , srv558346034:84 , srv577411026:84 , srv582719514:84 , srv608744596:84 , srv647174189:84 , srv651370133:84 , srv652296740:84 , srv656615194:84 , srv659163537:84 , srv660294019:84 , srv661971806:84 , srv741854101:84 , srv753218674:84 , srv771855682:84 , srv781880980:84 , srv782435758:84 , srv812183973:84 , srv869516859:84 , srv955840502:84 , srv973726335:84 , srv981804671:84 , srv120656823:85 , srv1220141811:85 , srv1320116900:85 , srv1574857613:85 , srv1728990370:85 , srv1884088825:85 , srv293971421:85 , srv663604624:85 , srv736053391:85 , srv849222312:85 , srv2090479042:683 }
2023-05-06 03:53:40,951 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=375 (was 338) - SystemLoadAverage LEAK? -, ProcessCount=179 (was 179), AvailableMemoryMB=4052 (was 4924)
2023-05-06 03:53:40,962 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=375, ProcessCount=179, AvailableMemoryMB=4050
2023-05-06 03:53:40,963 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-05-06 03:53:40,963 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc.
2023-05-06 03:53:40,975 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1131627041:84 , srv1134447433:84 , srv1194607813:84 , srv1294513841:84 , srv1301186544:84 , srv1381666646:84 , srv1546402608:84 , srv1584453575:84 , srv1606079186:84 , srv1663048155:84 , srv1743596650:84 , srv1765199659:84 , srv1995833335:84 , srv308027061:84 , srv376964005:84 , srv390410905:84 , srv441360216:84 , srv444066025:84 , srv464011857:84 , srv513644793:84 , srv608754702:84 , srv706738474:84 , srv708686115:84 , srv733317825:84 , srv80211661:84 , srv917321019:84 , srv95489719:84 , srv983518239:84 , srv989988930:84 , srv2140914393:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90]
2023-05-06 03:53:40,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-05-06 03:53:40,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1294513841=3, srv444066025=18, srv983518239=28, srv733317825=24, srv308027061=14, srv1546402608=6, srv917321019=26, srv1995833335=12, srv1381666646=5, srv390410905=16, srv80211661=25, srv1606079186=8, srv989988930=29, srv708686115=23, srv706738474=22, srv513644793=20, srv1301186544=4, srv1134447433=1, srv2140914393=13, srv95489719=27, srv1663048155=9, srv608754702=21, srv1765199659=11, srv441360216=17, srv1131627041=0, srv1743596650=10, srv1584453575=7, srv1194607813=2, srv376964005=15, srv464011857=19} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-05-06 03:53:40,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-05-06 03:53:40,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-05-06 03:53:40,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-05-06 03:53:40,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-05-06 03:53:40,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-05-06 03:53:40,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-05-06 03:53:40,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-05-06 03:53:40,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-05-06 03:53:40,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-05-06 03:53:40,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-05-06 03:53:40,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-05-06 03:53:40,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-05-06 03:53:40,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-05-06 03:53:40,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-05-06 03:53:40,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-05-06 03:53:40,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-05-06 03:53:40,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-05-06 03:53:40,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-05-06 03:53:40,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-05-06 03:53:40,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-05-06 03:53:40,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-05-06 03:53:40,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-05-06 03:53:40,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-05-06 03:53:40,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-05-06 03:53:40,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-05-06 03:53:40,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-05-06 03:53:40,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-05-06 03:53:40,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-05-06 03:53:40,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-05-06 03:53:40,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-05-06 03:53:40,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-05-06 03:53:41,015 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1463324001530617, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-06 03:53:41,015 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912654131878968, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1463324001530617, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=64800000
2023-05-06 03:55:40,996 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6942739 different iterations.  Found a solution that moves 1870 regions; Going from a computed imbalance of 0.9912654131878968 to a new imbalance of 4.3646732880327594E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6925925925925925, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-06 03:55:41,007 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1131627041:90 , srv1134447433:90 , srv1194607813:90 , srv1294513841:90 , srv1301186544:90 , srv1381666646:90 , srv1546402608:90 , srv1584453575:90 , srv1606079186:90 , srv1663048155:90 , srv1743596650:90 , srv1765199659:90 , srv1995833335:90 , srv2140914393:90 , srv308027061:90 , srv376964005:90 , srv390410905:90 , srv441360216:90 , srv444066025:90 , srv464011857:90 , srv513644793:90 , srv608754702:90 , srv706738474:90 , srv708686115:90 , srv733317825:90 , srv80211661:90 , srv917321019:90 , srv95489719:90 , srv983518239:90 , srv989988930:90 }
2023-05-06 03:55:41,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-05-06 03:55:41,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1294513841=3, srv444066025=18, srv983518239=28, srv733317825=24, srv308027061=14, srv1546402608=6, srv917321019=26, srv1995833335=12, srv1381666646=5, srv390410905=16, srv80211661=25, srv1606079186=8, srv989988930=29, srv708686115=23, srv706738474=22, srv513644793=20, srv1301186544=4, srv1134447433=1, srv2140914393=13, srv95489719=27, srv1663048155=9, srv608754702=21, srv1765199659=11, srv441360216=17, srv1131627041=0, srv1743596650=10, srv1584453575=7, srv1194607813=2, srv376964005=15, srv464011857=19} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-05-06 03:55:41,014 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-05-06 03:55:41,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-05-06 03:55:41,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-05-06 03:55:41,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-05-06 03:55:41,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-05-06 03:55:41,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-05-06 03:55:41,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-05-06 03:55:41,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-05-06 03:55:41,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-05-06 03:55:41,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-05-06 03:55:41,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-05-06 03:55:41,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-05-06 03:55:41,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-05-06 03:55:41,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-05-06 03:55:41,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-05-06 03:55:41,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-05-06 03:55:41,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-05-06 03:55:41,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-05-06 03:55:41,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-05-06 03:55:41,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-05-06 03:55:41,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-05-06 03:55:41,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-05-06 03:55:41,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-05-06 03:55:41,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-05-06 03:55:41,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-05-06 03:55:41,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-05-06 03:55:41,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-05-06 03:55:41,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-05-06 03:55:41,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-05-06 03:55:41,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-05-06 03:55:41,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-05-06 03:55:41,032 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-06 03:55:41,032 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1131627041:90 , srv1134447433:90 , srv1194607813:90 , srv1294513841:90 , srv1301186544:90 , srv1381666646:90 , srv1546402608:90 , srv1584453575:90 , srv1606079186:90 , srv1663048155:90 , srv1743596650:90 , srv1765199659:90 , srv1995833335:90 , srv2140914393:90 , srv308027061:90 , srv376964005:90 , srv390410905:90 , srv441360216:90 , srv444066025:90 , srv464011857:90 , srv513644793:90 , srv608754702:90 , srv706738474:90 , srv708686115:90 , srv733317825:90 , srv80211661:90 , srv917321019:90 , srv95489719:90 , srv983518239:90 , srv989988930:90 }
2023-05-06 03:55:41,048 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=9 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=191 (was 375), ProcessCount=179 (was 179), AvailableMemoryMB=4083 (was 4050) - AvailableMemoryMB LEAK? -