The screenshot of the tests results look like this:
We'll refer the average score after 10,944 matches as the "true score" and "Random Number Generation" as "RNG".
Please notice, for everyone's convenience the data is translated into "a typical 38 matches season".
After analyzing the data here's what we found:
If you test a tactic for 1 season ( 1 season x 38 matches = total 38 matches ): - The highest RNG = 25.8 points away from the "true score"
If you test a tactic for 2 seasons ( 2 seasons x 38 matches = total 76 matches ): - The highest RNG = 23.8 points away from the "true score"
If you test a tactic for 4 seasons ( 4 seasons x 38 matches = total 152 matches ): - The highest RNG = 11.6 points away from the "true score"
If you test a tactic for 8 seasons ( 8 seasons x 38 matches = total 304 matches ): - The highest RNG = 8.4 points away from the "true score"
If you test a tactic for 12 seasons ( 12 seasons x 38 matches = total 456 matches ): - The highest RNG = 6.6 points away from the "true score"
If you test a tactic for 24 seasons ( 24 seasons x 38 matches = total 912 matches ): - The highest RNG = 4.3 points away from the "true score"
If you test a tactic for 48 seasons ( 48 seasons x 38 matches = 1,824 matches ): - The highest RNG = 4.0 points away from the "true score"
If you test a tactic for 84 seasons ( 84 seasons x 38 matches = 3,192 matches ): - The highest RNG = 1.9 point away from the "true score"
If you test a tactic for 168 seasons ( 168 seasons x 38 matches = 6,384 matches ): - The highest RNG = 1 point away from the "true score"
As you can see if you test a tactic only for 1 season ( 38 matches ) and you hit the highest RNG then your result will be about 25.8 points away from the "true score" and if you want to test with an accuracy of 1 point then you need to test a tactic 168 seasons or 6,384 matches.
Of course, you can test a tactic only 1 season ( 38 matches ) and hit "the true result" but the probability of doing so is about 3% or so.
Please not, that our testing methodology eliminates many factors that increase the RNG:
- We set the morale and conditions of all players in the testing league to 100% and "freeze" it so the morale and conditions don't change at all. Please note, that's true for all the teams in the league the AI controlled teams and the human controlled teams.
- No player rotation happens. All the teams in the league use the same starting eleven all the time.
- No transfers can happen. AI managers can't be sacked.
If your testing methodology doesn't eliminate the factors above then the RNG in your tests will much higher than we got in our 10,944 matches test.
Please note, if you test tactics with a very strong team such as PSG in French league where your opponents are much weaker than your team then you the RNG will be very small because almost with any tactic you'll be getting 100% win rate the end of the season but don't be fooled by such low RNG, you need to understand when your team is much stronger than your opponents then you winning matches mostly due to having much better players than your opponents and the quality of your tactic doesn't make any significant difference and the opposite is also true when your team is much weaker than your opponents then you loosing matches mostly due to having much weaker players than your opponents and in this case your tactic also doesn't make any significant difference. So testing tactics with the strongest or the weakest team in the league isn't a good idea at all and you need something in the middle.
The screenshot of the tests results look like this:
The average score after 10,944 matches will be referred to the "true score" and "Random Number Generation" will be referred to "RNG"
Please notice, for everyone's convenience the data is translated into "a typical 38 matches season".
After analyzing the data here's what we found:
If you test a tactic for 1 season ( 1 season x 38 matches = total 38 matches ): - The highest RNG = 25.8 points away from the "true score"
If you test a tactic for 2 seasons ( 2 seasons x 38 matches = total 76 matches ): - The highest RNG = 23.8 points away from the "true score"
If you test a tactic for 4 seasons ( 4 seasons x 38 matches = total 152 matches ): - The highest RNG = 11.6 points away from the "true score"
If you test a tactic for 8 seasons ( 8 seasons x 38 matches = total 304 matches ): - The highest RNG = 8.4 points away from the "true score"
If you test a tactic for 12 seasons ( 12 seasons x 38 matches = total 456 matches ): - The highest RNG = 6.6 points away from the "true score"
If you test a tactic for 24 seasons ( 24 seasons x 38 matches = total 912 matches ): - The highest RNG = 4.3 points away from the "true score"
If you test a tactic for 48 seasons ( 48 seasons x 38 matches = 1,824 matches ): - The highest RNG = 3.2 points away from the "true score"
If you test a tactic for 84 seasons ( 84 seasons x 38 matches = 3,192 matches ): - The highest RNG = 1.9 point away from the "true score"
If you test a tactic for 168 seasons ( 168 seasons x 38 matches = 6,384 matches ): - The highest RNG = 1 point away from the "true score"
As you can see if you test a tactic only for 1 season ( 38 matches ) and you hit the highest RNG then your result will be about 25.8 points away from the "true score" and if you want to test with an accuracy of 1 point then you need to test a tactic 168 seasons or 6,384 matches.
Of course, you can test a tactic only 1 season ( 38 matches ) and hit "the true result" but the probability of doing so is about 3% or so.
Please not, that our testing methodology eliminates many factors that increase the RNG:
- We set the morale and conditions of all players in the testing league to 100% and "freeze" it so the morale and conditions don't change at all. That's true for the AI controlled teams and human controlled teams.
- All the teams in the league use the same starting eleven all the time. No rotation happens.
- No transfers happen. AI managers can't be sacked.
If your testing methodology doesn't eliminate the factors above then the RNG in your tests will much higher than what we got in our 10,944 matches test.
Also, if you test tactics with a very strong team such as PSG in French league where your opponents are much weaker than your team then you won't have any RNG at all because almost with any tactic you'll be getting about 110-114 pts at the end of the season but don't be fooled by such low RNG, you need to understand when your team is much stronger than your opponents then you winning matches mostly due to having much better players than your opponents and the quality of your tactic doesn't make any significant difference and when your team is much weaker than your opponents then you loosing matches mostly due to having much weaker players than your opponents and in this case your tactic also doesn't make any significant difference. So testing tactics with the strongest or the weakest team in the league isn't a good idea at all and you need something in the middle. Expand
Wow, you guys are really getting through the testing now you are up and running. Thanks a lot. The community really appreciates all you guys do.
Great work, very interesting results! Thank you so much for the hard work.
How does the RNG play out according to the points in the testing table? According to this post, there are a lot of RNG involved in testing for 640 matches (as tactic table). Right now, the nr. 1 tactic on the table and the nr. 10 tactic are 5 point apart. If we include RNG in the testing, all of the top 10 tactics might actually be the best? But we cannot know for certain before we test more games?
Am I onto somehting here, or does this not make sence?
kjordafen said: Great work, very interesting results! Thank you so much for the hard work.
How does the RNG play out according to the points in the testing table? According to this post, there are a lot of RNG involved in testing for 640 matches (as tactic table). Right now, the nr. 1 tactic on the table and the nr. 10 tactic are 5 point apart. If we include RNG in the testing, all of the top 10 tactics might actually be the best? But we cannot know for certain before we test more games?
Am I onto somehting here, or does this not make sence? Expand
I guess it'll be like it was in FM22, at some point later they we'll give the top tested tactics additional testing rounds to reach 3,000 matches
Droid said: I guess it'll be like it was in FM22, at some point later they we'll give the top tested tactics additional testing rounds to reach 3,000 matches Expand
yes, that's what we're planning to do in the future
At the moment our testing algorithm looks like this:
Every tactic gets 2 testing rounds by default ( 1 round is 320 matches so 2 rounds would be 640 matches ) then our testing server calculates the result and if after 2 first testing rounds the result is below 48 points then the test is over and the testing server uploads the result but if after 2 first testing rounds the result is above 48 points then it gets an additional testing round ( 1 round = 320 matches ) and after that additional testing round the testing server calculates the result again and if it hits below 48 points then the test is over and the testing server uploads the result but if it hits above 48 points then it gets another additional testing round and that continues up to maximum 9 rounds ( 2,880 matches ).
The purpose of the algorithm above is to keep the results at the top as accurate as possible and the same time consume the resources of our testing server at reasonable rate.
Please note, that we can "tune" the numbers in our testing algorithm at any moment depending on the situation. For example, at this moment we have reduced the maximum testing rounds from 9 rounds ( 2,880 matches ) to 6 rounds ( 1,920 matches ) because there are lot of tactics in the queue await to be tested at the moment.
Imho even 5 runs could be enough for first impact (1600) and if some tactics are "suspicious" of big bias rng could be simple tested later when the queque ain't that long. At least even the tactians can understand with resonable timing if what they doing is totaly wrong or totaly good.
Hey,
I'm sure that many of you wonder how many matches it requites testing a tactic to get an accurate result? So we've tested FM RNG and found an answer.
We tested the same tactic for 10,944 matches and measured the RNG on different distances.
Here's you can download the screenshots of the tests results - 10,944 Matches Test - Download
The screenshot of the tests results look like this:
We'll refer the average score after 10,944 matches as the "true score" and "Random Number Generation" as "RNG".
Please notice, for everyone's convenience the data is translated into "a typical 38 matches season".
After analyzing the data here's what we found:
If you test a tactic for 1 season ( 1 season x 38 matches = total 38 matches ):
- The highest RNG = 25.8 points away from the "true score"
If you test a tactic for 2 seasons ( 2 seasons x 38 matches = total 76 matches ):
- The highest RNG = 23.8 points away from the "true score"
If you test a tactic for 4 seasons ( 4 seasons x 38 matches = total 152 matches ):
- The highest RNG = 11.6 points away from the "true score"
If you test a tactic for 8 seasons ( 8 seasons x 38 matches = total 304 matches ):
- The highest RNG = 8.4 points away from the "true score"
If you test a tactic for 12 seasons ( 12 seasons x 38 matches = total 456 matches ):
- The highest RNG = 6.6 points away from the "true score"
If you test a tactic for 24 seasons ( 24 seasons x 38 matches = total 912 matches ):
- The highest RNG = 4.3 points away from the "true score"
If you test a tactic for 48 seasons ( 48 seasons x 38 matches = 1,824 matches ):
- The highest RNG = 4.0 points away from the "true score"
If you test a tactic for 84 seasons ( 84 seasons x 38 matches = 3,192 matches ):
- The highest RNG = 1.9 point away from the "true score"
If you test a tactic for 168 seasons ( 168 seasons x 38 matches = 6,384 matches ):
- The highest RNG = 1 point away from the "true score"
As you can see if you test a tactic only for 1 season ( 38 matches ) and you hit the highest RNG then your result will be about 25.8 points away from the "true score" and if you want to test with an accuracy of 1 point then you need to test a tactic 168 seasons or 6,384 matches.
Of course, you can test a tactic only 1 season ( 38 matches ) and hit "the true result" but the probability of doing so is about 3% or so.
Please not, that our testing methodology eliminates many factors that increase the RNG:
- We set the morale and conditions of all players in the testing league to 100% and "freeze" it so the morale and conditions don't change at all. Please note, that's true for all the teams in the league the AI controlled teams and the human controlled teams.
- No player rotation happens. All the teams in the league use the same starting eleven all the time.
- No transfers can happen. AI managers can't be sacked.
If your testing methodology doesn't eliminate the factors above then the RNG in your tests will much higher than we got in our 10,944 matches test.
Please note, if you test tactics with a very strong team such as PSG in French league where your opponents are much weaker than your team then you the RNG will be very small because almost with any tactic you'll be getting 100% win rate the end of the season but don't be fooled by such low RNG, you need to understand when your team is much stronger than your opponents then you winning matches mostly due to having much better players than your opponents and the quality of your tactic doesn't make any significant difference and the opposite is also true when your team is much weaker than your opponents then you loosing matches mostly due to having much weaker players than your opponents and in this case your tactic also doesn't make any significant difference. So testing tactics with the strongest or the weakest team in the league isn't a good idea at all and you need something in the middle.
Zippo said: Hey,
I'm sure that many of you wonder how many matches it requites testing a tactic to get an accurate result? So we've tested that and found an answer.
We tested one tactic 10,944 matches to measure the RNG on different distances.
Here's you can download the screenshots of the tests results - 10,944 Matches Test - Download
The screenshot of the tests results look like this:
The average score after 10,944 matches will be referred to the "true score" and "Random Number Generation" will be referred to "RNG"
Please notice, for everyone's convenience the data is translated into "a typical 38 matches season".
After analyzing the data here's what we found:
If you test a tactic for 1 season ( 1 season x 38 matches = total 38 matches ):
- The highest RNG = 25.8 points away from the "true score"
If you test a tactic for 2 seasons ( 2 seasons x 38 matches = total 76 matches ):
- The highest RNG = 23.8 points away from the "true score"
If you test a tactic for 4 seasons ( 4 seasons x 38 matches = total 152 matches ):
- The highest RNG = 11.6 points away from the "true score"
If you test a tactic for 8 seasons ( 8 seasons x 38 matches = total 304 matches ):
- The highest RNG = 8.4 points away from the "true score"
If you test a tactic for 12 seasons ( 12 seasons x 38 matches = total 456 matches ):
- The highest RNG = 6.6 points away from the "true score"
If you test a tactic for 24 seasons ( 24 seasons x 38 matches = total 912 matches ):
- The highest RNG = 4.3 points away from the "true score"
If you test a tactic for 48 seasons ( 48 seasons x 38 matches = 1,824 matches ):
- The highest RNG = 3.2 points away from the "true score"
If you test a tactic for 84 seasons ( 84 seasons x 38 matches = 3,192 matches ):
- The highest RNG = 1.9 point away from the "true score"
If you test a tactic for 168 seasons ( 168 seasons x 38 matches = 6,384 matches ):
- The highest RNG = 1 point away from the "true score"
As you can see if you test a tactic only for 1 season ( 38 matches ) and you hit the highest RNG then your result will be about 25.8 points away from the "true score" and if you want to test with an accuracy of 1 point then you need to test a tactic 168 seasons or 6,384 matches.
Of course, you can test a tactic only 1 season ( 38 matches ) and hit "the true result" but the probability of doing so is about 3% or so.
Please not, that our testing methodology eliminates many factors that increase the RNG:
- We set the morale and conditions of all players in the testing league to 100% and "freeze" it so the morale and conditions don't change at all. That's true for the AI controlled teams and human controlled teams.
- All the teams in the league use the same starting eleven all the time. No rotation happens.
- No transfers happen. AI managers can't be sacked.
If your testing methodology doesn't eliminate the factors above then the RNG in your tests will much higher than what we got in our 10,944 matches test.
Also, if you test tactics with a very strong team such as PSG in French league where your opponents are much weaker than your team then you won't have any RNG at all because almost with any tactic you'll be getting about 110-114 pts at the end of the season but don't be fooled by such low RNG, you need to understand when your team is much stronger than your opponents then you winning matches mostly due to having much better players than your opponents and the quality of your tactic doesn't make any significant difference and when your team is much weaker than your opponents then you loosing matches mostly due to having much weaker players than your opponents and in this case your tactic also doesn't make any significant difference. So testing tactics with the strongest or the weakest team in the league isn't a good idea at all and you need something in the middle.
Wow, you guys are really getting through the testing now you are up and running. Thanks a lot. The community really appreciates all you guys do.
Great work, very interesting results! Thank you so much for the hard work.
How does the RNG play out according to the points in the testing table? According to this post, there are a lot of RNG involved in testing for 640 matches (as tactic table). Right now, the nr. 1 tactic on the table and the nr. 10 tactic are 5 point apart. If we include RNG in the testing, all of the top 10 tactics might actually be the best? But we cannot know for certain before we test more games?
Am I onto somehting here, or does this not make sence?
kjordafen said: Great work, very interesting results! Thank you so much for the hard work.
How does the RNG play out according to the points in the testing table? According to this post, there are a lot of RNG involved in testing for 640 matches (as tactic table). Right now, the nr. 1 tactic on the table and the nr. 10 tactic are 5 point apart. If we include RNG in the testing, all of the top 10 tactics might actually be the best? But we cannot know for certain before we test more games?
Am I onto somehting here, or does this not make sence?
I guess it'll be like it was in FM22, at some point later they we'll give the top tested tactics additional testing rounds to reach 3,000 matches
FM arena is the only testing site I trust. You guys rock, keep up the great work.
Droid said: I guess it'll be like it was in FM22, at some point later they we'll give the top tested tactics additional testing rounds to reach 3,000 matches
yes, that's what we're planning to do in the future
Zippo said: yes, that's what we're planning to do in the future
Zippo, any news about attributes testing ? I'm curious about this, even if you think it's very close to 22*results
Falbravv said: Zippo, any news about attributes testing ? I'm curious about this, even if you think it's very close to 22*results
We're going to start testing the attributes soon, unfortunately, I can't give you any ETA.
Zippo said: We're going to start testing the attributes soon, unfortunately, I can't give you any ETA.
Obviously no problem, it's already a hard work doing here. Thanks
I'll put it here for more visibility.
At the moment our testing algorithm looks like this:
Every tactic gets 2 testing rounds by default ( 1 round is 320 matches so 2 rounds would be 640 matches ) then our testing server calculates the result and if after 2 first testing rounds the result is below 48 points then the test is over and the testing server uploads the result but if after 2 first testing rounds the result is above 48 points then it gets an additional testing round ( 1 round = 320 matches ) and after that additional testing round the testing server calculates the result again and if it hits below 48 points then the test is over and the testing server uploads the result but if it hits above 48 points then it gets another additional testing round and that continues up to maximum 9 rounds ( 2,880 matches ).
The purpose of the algorithm above is to keep the results at the top as accurate as possible and the same time consume the resources of our testing server at reasonable rate.
Please note, that we can "tune" the numbers in our testing algorithm at any moment depending on the situation. For example, at this moment we have reduced the maximum testing rounds from 9 rounds ( 2,880 matches ) to 6 rounds ( 1,920 matches ) because there are lot of tactics in the queue await to be tested at the moment.
Imho even 5 runs could be enough for first impact (1600) and if some tactics are "suspicious" of big bias rng could be simple tested later when the queque ain't that long. At least even the tactians can understand with resonable timing if what they doing is totaly wrong or totaly good.
Ofc are my 5cent