2023-04-08 02:56:46,717 INFO  [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins
2023-04-08 02:56:47,035 WARN  [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
2023-04-08 02:56:47,145 INFO  [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2023-04-08 02:56:47,203 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-04-08 02:56:47,210 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-04-08 02:56:47,223 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=341, ProcessCount=172, AvailableMemoryMB=2670
2023-04-08 02:56:47,225 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-04-08 02:56:47,225 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-04-08 02:56:47,364 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1014628102:84 , srv1023342061:84 , srv1035540781:84 , srv1060648615:84 , srv1066863595:84 , srv1084755976:84 , srv1109927544:84 , srv1141260519:84 , srv1145212544:84 , srv1156314881:84 , srv1236397457:84 , srv1263898995:84 , srv1272649876:84 , srv1276047144:84 , srv1288325212:84 , srv1301193283:84 , srv1329174691:84 , srv1337172465:84 , srv1343555636:84 , srv1352209510:84 , srv1380638032:84 , srv1416425387:84 , srv1471137359:84 , srv1476275067:84 , srv150436949:84 , srv1510779438:84 , srv1537189456:84 , srv1538242647:84 , srv1557889942:84 , srv1569409924:84 , srv1573748346:84 , srv1614955387:84 , srv1625035690:84 , srv1627433835:84 , srv1636502168:84 , srv1647344123:84 , srv1647677986:84 , srv1651071198:84 , srv1651096908:84 , srv174038650:84 , srv1744120379:84 , srv180895596:84 , srv1811855484:84 , srv1814119646:84 , srv181603872:84 , srv1818171569:84 , srv1819779837:84 , srv1828920282:84 , srv1840530232:84 , srv1843390825:84 , srv1859263467:84 , srv1894387807:84 , srv1923733802:84 , srv2016418885:84 , srv2030922060:84 , srv2033077991:84 , srv2061167159:84 , srv2062451575:84 , srv2064833803:84 , srv2082593251:84 , srv2120847229:84 , srv2138295387:84 , srv2143763591:84 , srv229734164:84 , srv245947434:84 , srv257798889:84 , srv328267704:84 , srv332342016:84 , srv334967354:84 , srv35652396:84 , srv360953834:84 , srv376851993:84 , srv410893377:84 , srv411850984:84 , srv426398117:84 , srv444764492:84 , srv489817665:84 , srv533801215:84 , srv535208220:84 , srv539835574:84 , srv553082158:84 , srv553131495:84 , srv560926095:84 , srv589382585:84 , srv637973610:84 , srv640223568:84 , srv660805969:84 , srv663623869:84 , srv691487780:84 , srv716443991:84 , srv764101575:84 , srv773884348:84 , srv780574923:84 , srv780614707:84 , srv791095946:84 , srv827378959:84 , srv878440150:84 , srv936668411:84 , srv958851785:84 , srv1200710744:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90]
2023-04-08 02:56:47,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-04-08 02:56:47,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1510779438=26, srv1843390825=50, srv245947434=65, srv1625035690=33, srv332342016=68, srv773884348=92, srv1538242647=28, srv1352209510=20, srv2016418885=54, srv1471137359=23, srv1647677986=37, srv2064833803=59, srv229734164=64, srv1023342061=1, srv1651071198=38, srv257798889=66, srv1200710744=10, srv764101575=91, srv1109927544=6, srv560926095=83, srv1651096908=39, srv1894387807=52, srv936668411=98, srv1337172465=18, srv444764492=76, srv1263898995=12, srv1276047144=14, srv2143763591=63, srv2061167159=57, srv174038650=40, srv553082158=81, srv660805969=87, srv1236397457=11, srv411850984=74, srv1569409924=30, srv2138295387=62, srv1301193283=16, srv1380638032=21, srv791095946=95, srv780574923=93, srv1814119646=44, srv1557889942=29, srv553131495=82, srv1614955387=32, srv640223568=86, srv827378959=96, srv180895596=42, srv2033077991=56, srv1859263467=51, srv663623869=88, srv780614707=94, srv716443991=90, srv2030922060=55, srv1014628102=0, srv1627433835=34, srv426398117=75, srv1156314881=9, srv1084755976=5, srv35652396=70, srv691487780=89, srv1636502168=35, srv1923733802=53, srv589382585=84, srv1272649876=13, srv1329174691=17, srv535208220=79, srv1035540781=2, srv1744120379=41, srv2082593251=60, srv539835574=80, srv181603872=45, srv637973610=85, srv1818171569=46, srv410893377=73, srv1416425387=22, srv334967354=69, srv533801215=78, srv1145212544=8, srv328267704=67, srv1537189456=27, srv376851993=72, srv1343555636=19, srv1476275067=24, srv1828920282=48, srv1066863595=4, srv360953834=71, srv1819779837=47, srv2120847229=61, srv958851785=99, srv1647344123=36, srv1811855484=43, srv489817665=77, srv2062451575=58, srv1288325212=15, srv150436949=25, srv1573748346=31, srv878440150=97, srv1060648615=3, srv1840530232=49, srv1141260519=7} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-04-08 02:56:47,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-04-08 02:56:47,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-04-08 02:56:47,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-04-08 02:56:47,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-04-08 02:56:47,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-04-08 02:56:47,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-04-08 02:56:47,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-04-08 02:56:47,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-04-08 02:56:47,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-04-08 02:56:47,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-04-08 02:56:47,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-04-08 02:56:47,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-04-08 02:56:47,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-04-08 02:56:47,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-04-08 02:56:47,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-04-08 02:56:47,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-04-08 02:56:47,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-04-08 02:56:47,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-04-08 02:56:47,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-04-08 02:56:47,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-04-08 02:56:47,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-04-08 02:56:47,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-04-08 02:56:47,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-04-08 02:56:47,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-04-08 02:56:47,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-04-08 02:56:47,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-04-08 02:56:47,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-04-08 02:56:47,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-04-08 02:56:47,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-04-08 02:56:47,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-04-08 02:56:47,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-04-08 02:56:47,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-04-08 02:56:47,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-04-08 02:56:47,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-04-08 02:56:47,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-04-08 02:56:47,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-04-08 02:56:47,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-04-08 02:56:47,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-04-08 02:56:47,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-04-08 02:56:47,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-04-08 02:56:47,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-04-08 02:56:47,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-04-08 02:56:47,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-04-08 02:56:47,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-04-08 02:56:47,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-04-08 02:56:47,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-04-08 02:56:47,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-04-08 02:56:47,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-04-08 02:56:47,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-04-08 02:56:47,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-04-08 02:56:47,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-04-08 02:56:47,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-04-08 02:56:47,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-04-08 02:56:47,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-04-08 02:56:47,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-04-08 02:56:47,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-04-08 02:56:47,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-04-08 02:56:47,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-04-08 02:56:47,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-04-08 02:56:47,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-04-08 02:56:47,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-04-08 02:56:47,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-04-08 02:56:47,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-04-08 02:56:47,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-04-08 02:56:47,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-04-08 02:56:47,459 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-04-08 02:56:47,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-04-08 02:56:47,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-04-08 02:56:47,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-04-08 02:56:47,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-04-08 02:56:47,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-04-08 02:56:47,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-04-08 02:56:47,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-04-08 02:56:47,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-04-08 02:56:47,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-04-08 02:56:47,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-04-08 02:56:47,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-04-08 02:56:47,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-04-08 02:56:47,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-04-08 02:56:47,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-04-08 02:56:47,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-04-08 02:56:47,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-04-08 02:56:47,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-04-08 02:56:47,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-04-08 02:56:47,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-04-08 02:56:47,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-04-08 02:56:47,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-04-08 02:56:47,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-04-08 02:56:47,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-04-08 02:56:47,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-04-08 02:56:47,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-04-08 02:56:47,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-04-08 02:56:47,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-04-08 02:56:47,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-04-08 02:56:47,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-04-08 02:56:47,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-04-08 02:56:47,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-04-08 02:56:47,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-04-08 02:56:47,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-04-08 02:56:47,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-04-08 02:56:47,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-04-08 02:56:47,658 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-08 02:56:47,659 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-04-08 02:56:47,659 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-04-08 02:56:48,286 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 791 ms to try 5000 different iterations.  Found a solution that moves 4664 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.18450127557254684. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04018134475589783, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06078055687698241, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5182222222222223, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04018134475589783, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.18, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24383333333333335, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-08 02:56:48,307 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv535208220:81 , srv1066863595:82 , srv1811855484:82 , srv328267704:82 , srv334967354:82 , srv1288325212:83 , srv1510779438:83 , srv1647344123:83 , srv35652396:83 , srv360953834:83 , srv539835574:83 , srv637973610:83 , srv660805969:83 , srv1109927544:84 , srv1352209510:84 , srv1894387807:84 , srv2082593251:84 , srv2120847229:84 , srv257798889:84 , srv426398117:84 , srv489817665:84 , srv716443991:84 , srv1060648615:85 , srv1236397457:85 , srv1263898995:85 , srv1272649876:85 , srv1301193283:85 , srv1476275067:85 , srv1538242647:85 , srv1557889942:85 , srv1569409924:85 , srv1573748346:85 , srv1614955387:85 , srv1744120379:85 , srv1814119646:85 , srv2016418885:85 , srv2138295387:85 , srv2143763591:85 , srv410893377:85 , srv444764492:85 , srv791095946:85 , srv958851785:85 , srv1014628102:86 , srv1084755976:86 , srv1329174691:86 , srv1380638032:86 , srv1537189456:86 , srv1651096908:86 , srv1818171569:86 , srv1828920282:86 , srv1859263467:86 , srv1923733802:86 , srv533801215:86 , srv663623869:86 , srv764101575:86 , srv780614707:86 , srv827378959:86 , srv878440150:86 , srv1337172465:87 , srv180895596:87 , srv181603872:87 , srv2061167159:87 , srv2062451575:87 , srv2064833803:87 , srv229734164:87 , srv553082158:87 , srv589382585:87 , srv640223568:87 , srv1023342061:88 , srv1035540781:88 , srv1145212544:88 , srv1156314881:88 , srv1471137359:88 , srv174038650:88 , srv1819779837:88 , srv2030922060:88 , srv2033077991:88 , srv245947434:88 , srv332342016:88 , srv411850984:88 , srv691487780:88 , srv780574923:88 , srv1625035690:89 , srv1636502168:89 , srv1647677986:89 , srv376851993:89 , srv1343555636:90 , srv1627433835:90 , srv1840530232:90 , srv553131495:90 , srv1141260519:91 , srv1276047144:91 , srv1416425387:91 , srv150436949:91 , srv1843390825:92 , srv936668411:92 , srv1651071198:93 , srv560926095:94 , srv773884348:95 , srv1200710744:447 }
2023-04-08 02:56:48,311 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-04-08 02:56:48,312 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1510779438=26, srv1843390825=50, srv245947434=65, srv1625035690=33, srv332342016=68, srv773884348=92, srv1538242647=28, srv1352209510=20, srv2016418885=54, srv1471137359=23, srv1647677986=37, srv2064833803=59, srv229734164=64, srv1023342061=1, srv1651071198=38, srv257798889=66, srv1200710744=10, srv764101575=91, srv1109927544=6, srv560926095=83, srv1651096908=39, srv1894387807=52, srv936668411=98, srv1337172465=18, srv444764492=76, srv1263898995=12, srv1276047144=14, srv2143763591=63, srv2061167159=57, srv174038650=40, srv553082158=81, srv660805969=87, srv1236397457=11, srv411850984=74, srv1569409924=30, srv2138295387=62, srv1301193283=16, srv1380638032=21, srv791095946=95, srv780574923=93, srv1814119646=44, srv1557889942=29, srv553131495=82, srv1614955387=32, srv640223568=86, srv827378959=96, srv180895596=42, srv2033077991=56, srv1859263467=51, srv663623869=88, srv780614707=94, srv716443991=90, srv2030922060=55, srv1014628102=0, srv1627433835=34, srv426398117=75, srv1156314881=9, srv1084755976=5, srv35652396=70, srv691487780=89, srv1636502168=35, srv1923733802=53, srv589382585=84, srv1272649876=13, srv1329174691=17, srv535208220=79, srv1035540781=2, srv1744120379=41, srv2082593251=60, srv539835574=80, srv181603872=45, srv637973610=85, srv1818171569=46, srv410893377=73, srv1416425387=22, srv334967354=69, srv533801215=78, srv1145212544=8, srv328267704=67, srv1537189456=27, srv376851993=72, srv1343555636=19, srv1476275067=24, srv1828920282=48, srv1066863595=4, srv360953834=71, srv1819779837=47, srv2120847229=61, srv958851785=99, srv1647344123=36, srv1811855484=43, srv489817665=77, srv2062451575=58, srv1288325212=15, srv150436949=25, srv1573748346=31, srv878440150=97, srv1060648615=3, srv1840530232=49, srv1141260519=7} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-04-08 02:56:48,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-04-08 02:56:48,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-04-08 02:56:48,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-04-08 02:56:48,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-04-08 02:56:48,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-04-08 02:56:48,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-04-08 02:56:48,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-04-08 02:56:48,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-04-08 02:56:48,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-04-08 02:56:48,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-04-08 02:56:48,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-04-08 02:56:48,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-04-08 02:56:48,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-04-08 02:56:48,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-04-08 02:56:48,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-04-08 02:56:48,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-04-08 02:56:48,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-04-08 02:56:48,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-04-08 02:56:48,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-04-08 02:56:48,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-04-08 02:56:48,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-04-08 02:56:48,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-04-08 02:56:48,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-04-08 02:56:48,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-04-08 02:56:48,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-04-08 02:56:48,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-04-08 02:56:48,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-04-08 02:56:48,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-04-08 02:56:48,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-04-08 02:56:48,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-04-08 02:56:48,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-04-08 02:56:48,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-04-08 02:56:48,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-04-08 02:56:48,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-04-08 02:56:48,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-04-08 02:56:48,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-04-08 02:56:48,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-04-08 02:56:48,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-04-08 02:56:48,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-04-08 02:56:48,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-04-08 02:56:48,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-04-08 02:56:48,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-04-08 02:56:48,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-04-08 02:56:48,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-04-08 02:56:48,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-04-08 02:56:48,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-04-08 02:56:48,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-04-08 02:56:48,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-04-08 02:56:48,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-04-08 02:56:48,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-04-08 02:56:48,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-04-08 02:56:48,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-04-08 02:56:48,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-04-08 02:56:48,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-04-08 02:56:48,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-04-08 02:56:48,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-04-08 02:56:48,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-04-08 02:56:48,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-04-08 02:56:48,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-04-08 02:56:48,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-04-08 02:56:48,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-04-08 02:56:48,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-04-08 02:56:48,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-04-08 02:56:48,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-04-08 02:56:48,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-04-08 02:56:48,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-04-08 02:56:48,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-04-08 02:56:48,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-04-08 02:56:48,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-04-08 02:56:48,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-04-08 02:56:48,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-04-08 02:56:48,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-04-08 02:56:48,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-04-08 02:56:48,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-04-08 02:56:48,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-04-08 02:56:48,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-04-08 02:56:48,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-04-08 02:56:48,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-04-08 02:56:48,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-04-08 02:56:48,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-04-08 02:56:48,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-04-08 02:56:48,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-04-08 02:56:48,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-04-08 02:56:48,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-04-08 02:56:48,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-04-08 02:56:48,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-04-08 02:56:48,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-04-08 02:56:48,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-04-08 02:56:48,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-04-08 02:56:48,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-04-08 02:56:48,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-04-08 02:56:48,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-04-08 02:56:48,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-04-08 02:56:48,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-04-08 02:56:48,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-04-08 02:56:48,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-04-08 02:56:48,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-04-08 02:56:48,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-04-08 02:56:48,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-04-08 02:56:48,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-04-08 02:56:48,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-04-08 02:56:48,486 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04018134475589783, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06078055687698241, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04018134475589783, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.18, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24383333333333335, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-08 02:56:48,486 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-04-08 02:56:48,487 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.18446861754653282, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04018134475589783, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06078055687698241, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04018134475589783, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.18, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24383333333333335, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-04-08 02:56:48,803 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 445 ms to try 5000 different iterations.  Found a solution that moves 2624 regions; Going from a computed imbalance of 0.18446861754653282 to a new imbalance of 0.017384662608909824. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0212626729154325); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03690616608912972, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.29155555555555557, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0212626729154325); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.014416666666666666); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04575, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-08 02:56:48,813 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv773884348:77 , srv1141260519:79 , srv150436949:79 , srv1651071198:79 , srv1843390825:80 , srv560926095:80 , srv1343555636:81 , srv936668411:81 , srv1023342061:82 , srv1276047144:82 , srv1035540781:83 , srv1145212544:83 , srv1416425387:83 , srv1636502168:83 , srv1840530232:83 , srv245947434:83 , srv411850984:83 , srv780574923:83 , srv1627433835:84 , srv1647677986:84 , srv174038650:84 , srv180895596:84 , srv181603872:84 , srv1819779837:84 , srv2033077991:84 , srv2061167159:84 , srv2062451575:84 , srv553131495:84 , srv691487780:84 , srv1084755976:85 , srv1380638032:85 , srv1471137359:85 , srv1537189456:85 , srv1557889942:85 , srv1625035690:85 , srv1651096908:85 , srv1923733802:85 , srv2064833803:85 , srv2138295387:85 , srv332342016:85 , srv640223568:85 , srv827378959:85 , srv1060648615:86 , srv1156314881:86 , srv1329174691:86 , srv1337172465:86 , srv1538242647:86 , srv1614955387:86 , srv1744120379:86 , srv1818171569:86 , srv2016418885:86 , srv2030922060:86 , srv2143763591:86 , srv376851993:86 , srv410893377:86 , srv489817665:86 , srv553082158:86 , srv589382585:86 , srv958851785:86 , srv1014628102:87 , srv1272649876:87 , srv1476275067:87 , srv1569409924:87 , srv1573748346:87 , srv1814119646:87 , srv1828920282:87 , srv1859263467:87 , srv360953834:87 , srv426398117:87 , srv539835574:87 , srv663623869:87 , srv764101575:87 , srv1066863595:88 , srv1109927544:88 , srv1263898995:88 , srv1301193283:88 , srv1352209510:88 , srv1894387807:88 , srv2120847229:88 , srv229734164:88 , srv257798889:88 , srv328267704:88 , srv35652396:88 , srv444764492:88 , srv660805969:88 , srv716443991:88 , srv780614707:88 , srv1236397457:89 , srv1510779438:89 , srv2082593251:89 , srv533801215:89 , srv637973610:89 , srv878440150:89 , srv1288325212:90 , srv1647344123:90 , srv1811855484:90 , srv791095946:90 , srv334967354:92 , srv535208220:92 , srv1200710744:516 }
2023-04-08 02:56:48,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-04-08 02:56:48,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1510779438=26, srv1843390825=50, srv245947434=65, srv1625035690=33, srv332342016=68, srv773884348=92, srv1538242647=28, srv1352209510=20, srv2016418885=54, srv1471137359=23, srv1647677986=37, srv2064833803=59, srv229734164=64, srv1023342061=1, srv1651071198=38, srv257798889=66, srv1200710744=10, srv764101575=91, srv1109927544=6, srv560926095=83, srv1651096908=39, srv1894387807=52, srv936668411=98, srv1337172465=18, srv444764492=76, srv1263898995=12, srv1276047144=14, srv2143763591=63, srv2061167159=57, srv174038650=40, srv553082158=81, srv660805969=87, srv1236397457=11, srv411850984=74, srv1569409924=30, srv2138295387=62, srv1301193283=16, srv1380638032=21, srv791095946=95, srv780574923=93, srv1814119646=44, srv1557889942=29, srv553131495=82, srv1614955387=32, srv640223568=86, srv827378959=96, srv180895596=42, srv2033077991=56, srv1859263467=51, srv663623869=88, srv780614707=94, srv716443991=90, srv2030922060=55, srv1014628102=0, srv1627433835=34, srv426398117=75, srv1156314881=9, srv1084755976=5, srv35652396=70, srv691487780=89, srv1636502168=35, srv1923733802=53, srv589382585=84, srv1272649876=13, srv1329174691=17, srv535208220=79, srv1035540781=2, srv1744120379=41, srv2082593251=60, srv539835574=80, srv181603872=45, srv637973610=85, srv1818171569=46, srv410893377=73, srv1416425387=22, srv334967354=69, srv533801215=78, srv1145212544=8, srv328267704=67, srv1537189456=27, srv376851993=72, srv1343555636=19, srv1476275067=24, srv1828920282=48, srv1066863595=4, srv360953834=71, srv1819779837=47, srv2120847229=61, srv958851785=99, srv1647344123=36, srv1811855484=43, srv489817665=77, srv2062451575=58, srv1288325212=15, srv150436949=25, srv1573748346=31, srv878440150=97, srv1060648615=3, srv1840530232=49, srv1141260519=7} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-04-08 02:56:48,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-04-08 02:56:48,850 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-04-08 02:56:48,850 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-04-08 02:56:48,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-04-08 02:56:48,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-04-08 02:56:48,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-04-08 02:56:48,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-04-08 02:56:48,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-04-08 02:56:48,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-04-08 02:56:48,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-04-08 02:56:48,852 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-04-08 02:56:48,852 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-04-08 02:56:48,852 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-04-08 02:56:48,852 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-04-08 02:56:48,852 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-04-08 02:56:48,852 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-04-08 02:56:48,852 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-04-08 02:56:48,852 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-04-08 02:56:48,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-04-08 02:56:48,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-04-08 02:56:48,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-04-08 02:56:48,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-04-08 02:56:48,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-04-08 02:56:48,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-04-08 02:56:48,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-04-08 02:56:48,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-04-08 02:56:48,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-04-08 02:56:48,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-04-08 02:56:48,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-04-08 02:56:48,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-04-08 02:56:48,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-04-08 02:56:48,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-04-08 02:56:48,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-04-08 02:56:48,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-04-08 02:56:48,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-04-08 02:56:48,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-04-08 02:56:48,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-04-08 02:56:48,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-04-08 02:56:48,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-04-08 02:56:48,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-04-08 02:56:48,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-04-08 02:56:48,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-04-08 02:56:48,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-04-08 02:56:48,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-04-08 02:56:48,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-04-08 02:56:48,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-04-08 02:56:48,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-04-08 02:56:48,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-04-08 02:56:48,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-04-08 02:56:48,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-04-08 02:56:48,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-04-08 02:56:48,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-04-08 02:56:48,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-04-08 02:56:48,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-04-08 02:56:48,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-04-08 02:56:48,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-04-08 02:56:48,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-04-08 02:56:48,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-04-08 02:56:48,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-04-08 02:56:48,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-04-08 02:56:48,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-04-08 02:56:48,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-04-08 02:56:48,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-04-08 02:56:48,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-04-08 02:56:48,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-04-08 02:56:48,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-04-08 02:56:48,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-04-08 02:56:48,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-04-08 02:56:48,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-04-08 02:56:48,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-04-08 02:56:48,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-04-08 02:56:48,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-04-08 02:56:48,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-04-08 02:56:48,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-04-08 02:56:48,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-04-08 02:56:48,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-04-08 02:56:48,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-04-08 02:56:48,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-04-08 02:56:48,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-04-08 02:56:48,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-04-08 02:56:48,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-04-08 02:56:48,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-04-08 02:56:48,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-04-08 02:56:48,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-04-08 02:56:48,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-04-08 02:56:48,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-04-08 02:56:48,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-04-08 02:56:48,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-04-08 02:56:48,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-04-08 02:56:48,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-04-08 02:56:48,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-04-08 02:56:48,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-04-08 02:56:48,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-04-08 02:56:48,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-04-08 02:56:48,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-04-08 02:56:48,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-04-08 02:56:48,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-04-08 02:56:48,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-04-08 02:56:48,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-04-08 02:56:48,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-04-08 02:56:48,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-04-08 02:56:48,999 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0212626729154325); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03690616608912972, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0212626729154325); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.014416666666666666); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04575, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-08 02:56:49,000 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-04-08 02:56:49,000 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.017366288968202127, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0212626729154325); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03690616608912972, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0212626729154325); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.014416666666666666); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04575, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-04-08 02:56:49,227 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 345 ms to try 5000 different iterations.  Found a solution that moves 1035 regions; Going from a computed imbalance of 0.017366288968202127 to a new imbalance of 5.930140452054307E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008030991224772667); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.011537281588060496); MoveCostFunction : (multiplier=7.0, imbalance=0.115, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.008030991224772667); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.6666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.003833333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-08 02:56:49,231 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv533801215:82 , srv780614707:82 , srv791095946:82 , srv878440150:82 , srv1156314881:83 , srv1236397457:83 , srv1416425387:83 , srv1627433835:83 , srv2030922060:83 , srv229734164:83 , srv376851993:83 , srv553131495:83 , srv764101575:83 , srv1014628102:84 , srv1109927544:84 , srv1263898995:84 , srv1276047144:84 , srv1288325212:84 , srv1301193283:84 , srv1471137359:84 , srv1569409924:84 , srv1625035690:84 , srv1647344123:84 , srv1811855484:84 , srv1814119646:84 , srv1828920282:84 , srv1859263467:84 , srv1894387807:84 , srv332342016:84 , srv334967354:84 , srv444764492:84 , srv535208220:84 , srv560926095:84 , srv663623869:84 , srv716443991:84 , srv1145212544:85 , srv1272649876:85 , srv1337172465:85 , srv1352209510:85 , srv1476275067:85 , srv1510779438:85 , srv1537189456:85 , srv1651071198:85 , srv1651096908:85 , srv1819779837:85 , srv1840530232:85 , srv2033077991:85 , srv2064833803:85 , srv2082593251:85 , srv553082158:85 , srv589382585:85 , srv640223568:85 , srv660805969:85 , srv827378959:85 , srv936668411:85 , srv1060648615:86 , srv1066863595:86 , srv1084755976:86 , srv1141260519:86 , srv1329174691:86 , srv1538242647:86 , srv1573748346:86 , srv1614955387:86 , srv1636502168:86 , srv174038650:86 , srv180895596:86 , srv1818171569:86 , srv1843390825:86 , srv1923733802:86 , srv2016418885:86 , srv2061167159:86 , srv2138295387:86 , srv2143763591:86 , srv257798889:86 , srv35652396:86 , srv360953834:86 , srv410893377:86 , srv411850984:86 , srv637973610:86 , srv773884348:86 , srv958851785:86 , srv1023342061:87 , srv1343555636:87 , srv1744120379:87 , srv181603872:87 , srv2062451575:87 , srv2120847229:87 , srv426398117:87 , srv489817665:87 , srv691487780:87 , srv780574923:87 , srv1380638032:88 , srv150436949:88 , srv1557889942:88 , srv1647677986:88 , srv245947434:88 , srv328267704:88 , srv539835574:88 , srv1035540781:89 , srv1200710744:566 }
2023-04-08 02:56:49,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-04-08 02:56:49,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1510779438=26, srv1843390825=50, srv245947434=65, srv1625035690=33, srv332342016=68, srv773884348=92, srv1538242647=28, srv1352209510=20, srv2016418885=54, srv1471137359=23, srv1647677986=37, srv2064833803=59, srv229734164=64, srv1023342061=1, srv1651071198=38, srv257798889=66, srv1200710744=10, srv764101575=91, srv1109927544=6, srv560926095=83, srv1651096908=39, srv1894387807=52, srv936668411=98, srv1337172465=18, srv444764492=76, srv1263898995=12, srv1276047144=14, srv2143763591=63, srv2061167159=57, srv174038650=40, srv553082158=81, srv660805969=87, srv1236397457=11, srv411850984=74, srv1569409924=30, srv2138295387=62, srv1301193283=16, srv1380638032=21, srv791095946=95, srv780574923=93, srv1814119646=44, srv1557889942=29, srv553131495=82, srv1614955387=32, srv640223568=86, srv827378959=96, srv180895596=42, srv2033077991=56, srv1859263467=51, srv663623869=88, srv780614707=94, srv716443991=90, srv2030922060=55, srv1014628102=0, srv1627433835=34, srv426398117=75, srv1156314881=9, srv1084755976=5, srv35652396=70, srv691487780=89, srv1636502168=35, srv1923733802=53, srv589382585=84, srv1272649876=13, srv1329174691=17, srv535208220=79, srv1035540781=2, srv1744120379=41, srv2082593251=60, srv539835574=80, srv181603872=45, srv637973610=85, srv1818171569=46, srv410893377=73, srv1416425387=22, srv334967354=69, srv533801215=78, srv1145212544=8, srv328267704=67, srv1537189456=27, srv376851993=72, srv1343555636=19, srv1476275067=24, srv1828920282=48, srv1066863595=4, srv360953834=71, srv1819779837=47, srv2120847229=61, srv958851785=99, srv1647344123=36, srv1811855484=43, srv489817665=77, srv2062451575=58, srv1288325212=15, srv150436949=25, srv1573748346=31, srv878440150=97, srv1060648615=3, srv1840530232=49, srv1141260519=7} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-04-08 02:56:49,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-04-08 02:56:49,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-04-08 02:56:49,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-04-08 02:56:49,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-04-08 02:56:49,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-04-08 02:56:49,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-04-08 02:56:49,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-04-08 02:56:49,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-04-08 02:56:49,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-04-08 02:56:49,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-04-08 02:56:49,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-04-08 02:56:49,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-04-08 02:56:49,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-04-08 02:56:49,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-04-08 02:56:49,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-04-08 02:56:49,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-04-08 02:56:49,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-04-08 02:56:49,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-04-08 02:56:49,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-04-08 02:56:49,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-04-08 02:56:49,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-04-08 02:56:49,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-04-08 02:56:49,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-04-08 02:56:49,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-04-08 02:56:49,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-04-08 02:56:49,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-04-08 02:56:49,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-04-08 02:56:49,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-04-08 02:56:49,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-04-08 02:56:49,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-04-08 02:56:49,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-04-08 02:56:49,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-04-08 02:56:49,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-04-08 02:56:49,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-04-08 02:56:49,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-04-08 02:56:49,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-04-08 02:56:49,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-04-08 02:56:49,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-04-08 02:56:49,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-04-08 02:56:49,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-04-08 02:56:49,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-04-08 02:56:49,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-04-08 02:56:49,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-04-08 02:56:49,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-04-08 02:56:49,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-04-08 02:56:49,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-04-08 02:56:49,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-04-08 02:56:49,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-04-08 02:56:49,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-04-08 02:56:49,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-04-08 02:56:49,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-04-08 02:56:49,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-04-08 02:56:49,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-04-08 02:56:49,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-04-08 02:56:49,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-04-08 02:56:49,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-04-08 02:56:49,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-04-08 02:56:49,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-04-08 02:56:49,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-04-08 02:56:49,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-04-08 02:56:49,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-04-08 02:56:49,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-04-08 02:56:49,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-04-08 02:56:49,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-04-08 02:56:49,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-04-08 02:56:49,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-04-08 02:56:49,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-04-08 02:56:49,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-04-08 02:56:49,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-04-08 02:56:49,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-04-08 02:56:49,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-04-08 02:56:49,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-04-08 02:56:49,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-04-08 02:56:49,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-04-08 02:56:49,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-04-08 02:56:49,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-04-08 02:56:49,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-04-08 02:56:49,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-04-08 02:56:49,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-04-08 02:56:49,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-04-08 02:56:49,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-04-08 02:56:49,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-04-08 02:56:49,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-04-08 02:56:49,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-04-08 02:56:49,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-04-08 02:56:49,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-04-08 02:56:49,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-04-08 02:56:49,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-04-08 02:56:49,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-04-08 02:56:49,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-04-08 02:56:49,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-04-08 02:56:49,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-04-08 02:56:49,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-04-08 02:56:49,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-04-08 02:56:49,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-04-08 02:56:49,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-04-08 02:56:49,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-04-08 02:56:49,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-04-08 02:56:49,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-04-08 02:56:49,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-04-08 02:56:49,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-04-08 02:56:49,401 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008030991224772667); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.011537281588060496); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.008030991224772667); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.6666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.003833333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-08 02:56:49,401 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-04-08 02:56:49,402 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=5.85766820307387E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008030991224772667); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.011537281588060496); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.008030991224772667); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.6666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.003833333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-04-08 02:56:49,722 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 444 ms to try 5000 different iterations.  Found a solution that moves 294 regions; Going from a computed imbalance of 5.85766820307387E-4 to a new imbalance of 2.0869482037348015E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.178341294946416E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0020651563939657444); MoveCostFunction : (multiplier=7.0, imbalance=0.03266666666666666, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.178341294946416E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-08 02:56:49,723 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1035540781:82 , srv1647677986:82 , srv1380638032:83 , srv2120847229:83 , srv245947434:83 , srv691487780:83 , srv1329174691:84 , srv1337172465:84 , srv1343555636:84 , srv150436949:84 , srv1557889942:84 , srv1573748346:84 , srv1625035690:84 , srv1636502168:84 , srv174038650:84 , srv181603872:84 , srv1818171569:84 , srv1819779837:84 , srv1843390825:84 , srv2016418885:84 , srv2062451575:84 , srv2082593251:84 , srv229734164:84 , srv257798889:84 , srv328267704:84 , srv334967354:84 , srv376851993:84 , srv426398117:84 , srv539835574:84 , srv553082158:84 , srv560926095:84 , srv589382585:84 , srv637973610:84 , srv773884348:84 , srv936668411:84 , srv1014628102:85 , srv1023342061:85 , srv1060648615:85 , srv1084755976:85 , srv1141260519:85 , srv1145212544:85 , srv1156314881:85 , srv1236397457:85 , srv1263898995:85 , srv1276047144:85 , srv1288325212:85 , srv1301193283:85 , srv1352209510:85 , srv1416425387:85 , srv1476275067:85 , srv1510779438:85 , srv1538242647:85 , srv1614955387:85 , srv1627433835:85 , srv1647344123:85 , srv1651071198:85 , srv1744120379:85 , srv180895596:85 , srv1828920282:85 , srv1840530232:85 , srv1859263467:85 , srv1923733802:85 , srv2030922060:85 , srv2033077991:85 , srv2064833803:85 , srv2143763591:85 , srv332342016:85 , srv35652396:85 , srv410893377:85 , srv411850984:85 , srv444764492:85 , srv489817665:85 , srv533801215:85 , srv535208220:85 , srv553131495:85 , srv640223568:85 , srv660805969:85 , srv663623869:85 , srv716443991:85 , srv764101575:85 , srv780574923:85 , srv791095946:85 , srv878440150:85 , srv958851785:85 , srv1066863595:86 , srv1109927544:86 , srv1272649876:86 , srv1471137359:86 , srv1537189456:86 , srv1569409924:86 , srv1651096908:86 , srv1811855484:86 , srv1814119646:86 , srv1894387807:86 , srv2061167159:86 , srv2138295387:86 , srv360953834:86 , srv780614707:86 , srv827378959:86 , srv1200710744:613 }
2023-04-08 02:56:49,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-04-08 02:56:49,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1510779438=26, srv1843390825=50, srv245947434=65, srv1625035690=33, srv332342016=68, srv773884348=92, srv1538242647=28, srv1352209510=20, srv2016418885=54, srv1471137359=23, srv1647677986=37, srv2064833803=59, srv229734164=64, srv1023342061=1, srv1651071198=38, srv257798889=66, srv1200710744=10, srv764101575=91, srv1109927544=6, srv560926095=83, srv1651096908=39, srv1894387807=52, srv936668411=98, srv1337172465=18, srv444764492=76, srv1263898995=12, srv1276047144=14, srv2143763591=63, srv2061167159=57, srv174038650=40, srv553082158=81, srv660805969=87, srv1236397457=11, srv411850984=74, srv1569409924=30, srv2138295387=62, srv1301193283=16, srv1380638032=21, srv791095946=95, srv780574923=93, srv1814119646=44, srv1557889942=29, srv553131495=82, srv1614955387=32, srv640223568=86, srv827378959=96, srv180895596=42, srv2033077991=56, srv1859263467=51, srv663623869=88, srv780614707=94, srv716443991=90, srv2030922060=55, srv1014628102=0, srv1627433835=34, srv426398117=75, srv1156314881=9, srv1084755976=5, srv35652396=70, srv691487780=89, srv1636502168=35, srv1923733802=53, srv589382585=84, srv1272649876=13, srv1329174691=17, srv535208220=79, srv1035540781=2, srv1744120379=41, srv2082593251=60, srv539835574=80, srv181603872=45, srv637973610=85, srv1818171569=46, srv410893377=73, srv1416425387=22, srv334967354=69, srv533801215=78, srv1145212544=8, srv328267704=67, srv1537189456=27, srv376851993=72, srv1343555636=19, srv1476275067=24, srv1828920282=48, srv1066863595=4, srv360953834=71, srv1819779837=47, srv2120847229=61, srv958851785=99, srv1647344123=36, srv1811855484=43, srv489817665=77, srv2062451575=58, srv1288325212=15, srv150436949=25, srv1573748346=31, srv878440150=97, srv1060648615=3, srv1840530232=49, srv1141260519=7} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-04-08 02:56:49,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-04-08 02:56:49,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-04-08 02:56:49,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-04-08 02:56:49,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-04-08 02:56:49,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-04-08 02:56:49,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-04-08 02:56:49,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-04-08 02:56:49,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-04-08 02:56:49,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-04-08 02:56:49,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-04-08 02:56:49,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-04-08 02:56:49,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-04-08 02:56:49,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-04-08 02:56:49,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-04-08 02:56:49,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-04-08 02:56:49,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-04-08 02:56:49,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-04-08 02:56:49,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-04-08 02:56:49,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-04-08 02:56:49,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-04-08 02:56:49,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-04-08 02:56:49,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-04-08 02:56:49,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-04-08 02:56:49,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-04-08 02:56:49,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-04-08 02:56:49,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-04-08 02:56:49,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-04-08 02:56:49,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-04-08 02:56:49,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-04-08 02:56:49,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-04-08 02:56:49,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-04-08 02:56:49,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-04-08 02:56:49,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-04-08 02:56:49,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-04-08 02:56:49,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-04-08 02:56:49,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-04-08 02:56:49,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-04-08 02:56:49,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-04-08 02:56:49,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-04-08 02:56:49,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-04-08 02:56:49,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-04-08 02:56:49,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-04-08 02:56:49,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-04-08 02:56:49,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-04-08 02:56:49,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-04-08 02:56:49,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-04-08 02:56:49,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-04-08 02:56:49,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-04-08 02:56:49,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-04-08 02:56:49,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-04-08 02:56:49,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-04-08 02:56:49,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-04-08 02:56:49,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-04-08 02:56:49,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-04-08 02:56:49,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-04-08 02:56:49,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-04-08 02:56:49,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-04-08 02:56:49,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-04-08 02:56:49,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-04-08 02:56:49,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-04-08 02:56:49,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-04-08 02:56:49,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-04-08 02:56:49,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-04-08 02:56:49,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-04-08 02:56:49,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-04-08 02:56:49,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-04-08 02:56:49,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-04-08 02:56:49,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-04-08 02:56:49,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-04-08 02:56:49,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-04-08 02:56:49,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-04-08 02:56:49,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-04-08 02:56:49,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-04-08 02:56:49,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-04-08 02:56:49,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-04-08 02:56:49,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-04-08 02:56:49,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-04-08 02:56:49,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-04-08 02:56:49,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-04-08 02:56:49,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-04-08 02:56:49,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-04-08 02:56:49,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-04-08 02:56:49,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-04-08 02:56:49,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-04-08 02:56:49,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-04-08 02:56:49,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-04-08 02:56:49,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-04-08 02:56:49,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-04-08 02:56:49,741 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-04-08 02:56:49,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-04-08 02:56:49,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-04-08 02:56:49,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-04-08 02:56:49,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-04-08 02:56:49,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-04-08 02:56:49,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-04-08 02:56:49,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-04-08 02:56:49,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-04-08 02:56:49,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-04-08 02:56:49,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-04-08 02:56:49,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-04-08 02:56:49,742 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-04-08 02:56:49,846 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=1.881085003732401E-5 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.178341294946416E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0020651563939657444); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.178341294946416E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-08 02:56:49,847 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1035540781:82 , srv1647677986:82 , srv1380638032:83 , srv2120847229:83 , srv245947434:83 , srv691487780:83 , srv1329174691:84 , srv1337172465:84 , srv1343555636:84 , srv150436949:84 , srv1557889942:84 , srv1573748346:84 , srv1625035690:84 , srv1636502168:84 , srv174038650:84 , srv181603872:84 , srv1818171569:84 , srv1819779837:84 , srv1843390825:84 , srv2016418885:84 , srv2062451575:84 , srv2082593251:84 , srv229734164:84 , srv257798889:84 , srv328267704:84 , srv334967354:84 , srv376851993:84 , srv426398117:84 , srv539835574:84 , srv553082158:84 , srv560926095:84 , srv589382585:84 , srv637973610:84 , srv773884348:84 , srv936668411:84 , srv1014628102:85 , srv1023342061:85 , srv1060648615:85 , srv1084755976:85 , srv1141260519:85 , srv1145212544:85 , srv1156314881:85 , srv1236397457:85 , srv1263898995:85 , srv1276047144:85 , srv1288325212:85 , srv1301193283:85 , srv1352209510:85 , srv1416425387:85 , srv1476275067:85 , srv1510779438:85 , srv1538242647:85 , srv1614955387:85 , srv1627433835:85 , srv1647344123:85 , srv1651071198:85 , srv1744120379:85 , srv180895596:85 , srv1828920282:85 , srv1840530232:85 , srv1859263467:85 , srv1923733802:85 , srv2030922060:85 , srv2033077991:85 , srv2064833803:85 , srv2143763591:85 , srv332342016:85 , srv35652396:85 , srv410893377:85 , srv411850984:85 , srv444764492:85 , srv489817665:85 , srv533801215:85 , srv535208220:85 , srv553131495:85 , srv640223568:85 , srv660805969:85 , srv663623869:85 , srv716443991:85 , srv764101575:85 , srv780574923:85 , srv791095946:85 , srv878440150:85 , srv958851785:85 , srv1066863595:86 , srv1109927544:86 , srv1272649876:86 , srv1471137359:86 , srv1537189456:86 , srv1569409924:86 , srv1651096908:86 , srv1811855484:86 , srv1814119646:86 , srv1894387807:86 , srv2061167159:86 , srv2138295387:86 , srv360953834:86 , srv780614707:86 , srv827378959:86 , srv1200710744:613 }
2023-04-08 02:56:49,873 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=341 (was 341), ProcessCount=172 (was 172), AvailableMemoryMB=2180 (was 2670)
2023-04-08 02:56:49,883 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=341, ProcessCount=172, AvailableMemoryMB=2180
2023-04-08 02:56:49,884 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-04-08 02:56:49,884 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc.
2023-04-08 02:56:49,894 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1004613844:84 , srv1029322594:84 , srv1065940810:84 , srv106993534:84 , srv1086124234:84 , srv1144668764:84 , srv1584513016:84 , srv1773125596:84 , srv1808069894:84 , srv1821417925:84 , srv1847101726:84 , srv1902521872:84 , srv1971875434:84 , srv2123191120:84 , srv242446495:84 , srv377341195:84 , srv382315483:84 , srv399643992:84 , srv494444598:84 , srv549652294:84 , srv571480888:84 , srv588045749:84 , srv617989290:84 , srv691936954:84 , srv770419210:84 , srv902713181:84 , srv925419607:84 , srv926441206:84 , srv93221026:84 , srv1217691517:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90]
2023-04-08 02:56:49,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-04-08 02:56:49,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1821417925=10, srv902713181=26, srv571480888=21, srv1004613844=0, srv1773125596=8, srv93221026=29, srv1144668764=5, srv399643992=18, srv1086124234=4, srv1808069894=9, srv1847101726=11, srv106993534=3, srv382315483=17, srv1902521872=12, srv494444598=19, srv617989290=23, srv1584513016=7, srv770419210=25, srv377341195=16, srv925419607=27, srv588045749=22, srv549652294=20, srv242446495=15, srv1065940810=2, srv1217691517=6, srv1971875434=13, srv1029322594=1, srv926441206=28, srv691936954=24, srv2123191120=14} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-04-08 02:56:49,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-04-08 02:56:49,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-04-08 02:56:49,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-04-08 02:56:49,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-04-08 02:56:49,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-04-08 02:56:49,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-04-08 02:56:49,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-04-08 02:56:49,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-04-08 02:56:49,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-04-08 02:56:49,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-04-08 02:56:49,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-04-08 02:56:49,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-04-08 02:56:49,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-04-08 02:56:49,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-04-08 02:56:49,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-04-08 02:56:49,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-04-08 02:56:49,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-04-08 02:56:49,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-04-08 02:56:49,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-04-08 02:56:49,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-04-08 02:56:49,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-04-08 02:56:49,904 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-04-08 02:56:49,904 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-04-08 02:56:49,904 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-04-08 02:56:49,904 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-04-08 02:56:49,904 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-04-08 02:56:49,904 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-04-08 02:56:49,904 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-04-08 02:56:49,904 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-04-08 02:56:49,904 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-04-08 02:56:49,904 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-04-08 02:56:49,918 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.140524260848357, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-08 02:56:49,918 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912635830621673, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.140524260848357, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=64800000
2023-04-08 02:58:49,907 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6559143 different iterations.  Found a solution that moves 1864 regions; Going from a computed imbalance of 0.9912635830621673 to a new imbalance of 4.3506689887128686E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6903703703703704, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-08 02:58:49,913 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1004613844:90 , srv1029322594:90 , srv1065940810:90 , srv106993534:90 , srv1086124234:90 , srv1144668764:90 , srv1217691517:90 , srv1584513016:90 , srv1773125596:90 , srv1808069894:90 , srv1821417925:90 , srv1847101726:90 , srv1902521872:90 , srv1971875434:90 , srv2123191120:90 , srv242446495:90 , srv377341195:90 , srv382315483:90 , srv399643992:90 , srv494444598:90 , srv549652294:90 , srv571480888:90 , srv588045749:90 , srv617989290:90 , srv691936954:90 , srv770419210:90 , srv902713181:90 , srv925419607:90 , srv926441206:90 , srv93221026:90 }
2023-04-08 02:58:49,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-04-08 02:58:49,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1821417925=10, srv902713181=26, srv571480888=21, srv1004613844=0, srv1773125596=8, srv93221026=29, srv1144668764=5, srv399643992=18, srv1086124234=4, srv1808069894=9, srv1847101726=11, srv106993534=3, srv382315483=17, srv1902521872=12, srv494444598=19, srv617989290=23, srv1584513016=7, srv770419210=25, srv377341195=16, srv925419607=27, srv588045749=22, srv549652294=20, srv242446495=15, srv1065940810=2, srv1217691517=6, srv1971875434=13, srv1029322594=1, srv926441206=28, srv691936954=24, srv2123191120=14} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-04-08 02:58:49,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-04-08 02:58:49,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-04-08 02:58:49,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-04-08 02:58:49,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-04-08 02:58:49,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-04-08 02:58:49,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-04-08 02:58:49,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-04-08 02:58:49,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-04-08 02:58:49,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-04-08 02:58:49,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-04-08 02:58:49,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-04-08 02:58:49,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-04-08 02:58:49,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-04-08 02:58:49,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-04-08 02:58:49,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-04-08 02:58:49,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-04-08 02:58:49,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-04-08 02:58:49,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-04-08 02:58:49,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-04-08 02:58:49,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-04-08 02:58:49,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-04-08 02:58:49,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-04-08 02:58:49,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-04-08 02:58:49,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-04-08 02:58:49,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-04-08 02:58:49,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-04-08 02:58:49,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-04-08 02:58:49,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-04-08 02:58:49,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-04-08 02:58:49,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-04-08 02:58:49,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-04-08 02:58:49,945 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-08 02:58:49,945 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1004613844:90 , srv1029322594:90 , srv1065940810:90 , srv106993534:90 , srv1086124234:90 , srv1144668764:90 , srv1217691517:90 , srv1584513016:90 , srv1773125596:90 , srv1808069894:90 , srv1821417925:90 , srv1847101726:90 , srv1902521872:90 , srv1971875434:90 , srv2123191120:90 , srv242446495:90 , srv377341195:90 , srv382315483:90 , srv399643992:90 , srv494444598:90 , srv549652294:90 , srv571480888:90 , srv588045749:90 , srv617989290:90 , srv691936954:90 , srv770419210:90 , srv902713181:90 , srv925419607:90 , srv926441206:90 , srv93221026:90 }
2023-04-08 02:58:49,960 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=9 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=343 (was 341) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 172), AvailableMemoryMB=2156 (was 2180)