CSTG KANE said: It's actually hard to confirm this, I can only guarantee that from now on, my tactical screenshots won't show the same Expand
Nope, pal. I'm afraid it won't work that way.
You have to check every of your tested tactic and make sure all of them have proper screenshots. If it's needed you have to run tests again to add proper screenshots.
Only when you can confirm that all your tested tactics and non-tested tactics have proper screenshots we start test your new tactics again.
dzek said: My brother perhaps I did not explain my question well. Certainly not making in-match subs will have an impact and the test will not be as reliable. My point is that maybe the 'Use of Subs' attribute has something to do with when these subs are made and if they are made in general.
An example: - The manager with 'Use of Subs' value 1 may not make subs in-match or may make them quite late - for example in the 85th minute. - The manager with 'Use of Subs' value 20 to make subs very quickly and in every game.
This again would have an effect on the results as a more rested player in the game would have their ability at full capacity while those who are tired would be -3/-5 in every attribute due to fatigue. Expand
Only the game devs know for sure how "Use of Subs" attribute works, we can only guess about it.
Anyway, regardless of its working mechanic, it would be the same for every tactic tested so every tactic tested is effected by it at the same degree and we're comparing tactics here.
dzek said: 1. It says here that within the testing league no rotation happens for any team (including AI teams) however this seems to be something that is not the case and I wanted to ask if squad rotation of AI teams affects/increases the RNG? Expand Hi,
Yes, the AI managers rotate their teams but the alternative players for every position have the same attributes, morale and conditions as the main players, the alternative players are just copies of the main players so it can be considered as "no rotations happens"
dzek said: 2. Within matches the AI managers and assistant managers (of user controlled teams) do players subs but the 'Use of Subs' attribute is different for each of them individually. Does this play a role in the sub strategy and in the results and if so, does this also affect the RNG?
Thank you! Expand
Subs must happen and restricting them would have unpredictable consequences for the reliability of the results.
CBP87 said: @Zippo@Droid do you have this information available please? is it possible to add the result of 1200/2400/4000 (if applicable) to the results table? Expand
Hi,
Unfortunately, I don't think something like that would be possible.
@juwow, thank you for your vigilance and contribution, it's much appreciated.
@A Smile, @dzek, @letsgo9, thank you for your opinions and involvement in this situation, it's much appreciated.
@CSTG KANE, I see only one fair solution to this situation: you have to fix all the threads that have the same screenshots and add correct ones. Until then non of your tactics will be tested. Let me know when you're done.
infxamus said: Is it possible that there is a game file of the save at the end of the season? There used to be game saves available to see after a test was completed Expand
Hi,
We dropped that feature because the demand for game saves was low and the cost of hosting them was too high.
But we still keep all the game saves in an offline storage and I can provide them by a request.
There seems to be some obvious changes from last year, such as high LOE + Low def line being the default option last year, but now everything is much higher + much higher.
Sorry if it's been asked before. Expand
The last year TIs testing was more like an experiment and during the process it became clear that the way TIs work depends on formation/roles/duties/mentality and so on... other words, if something works well in one tactic then it isn't necessary also work well in other tactics.
The above means that if you want to test the TIs in a proper way then you have to test all possible combinations of TIs/Mentality/Formations/Roles/Duties. That would be something unrealizable to do. Also, add to it retesting with patches.
Mark said: As I said, I only used the first 6 tests, so is your data showing the average from all tests where you have tested 4000 matches or only the first 2400 matches? Expand
The thing is, the order in which the result screenshots are displayed isn't the order in which the tests happened. The result screenshot are randomly shuffled.
Mark said: I also only looked at the first 6 tests (ie 2400 matches) for each HOF tactic to ensure they were also all comparable. Expand
As you can see we've got 1,200 matches, 2,400 matches and 4,000 matches in our test result tables and we compare all them just fine.
You just need to calculate P.P.M.(Points Per Match) and then translate it for 38 matches.
Total Points / Total Matches Played = P.P.M.(Points Per Match)
P.P.M.(Points Per Match) x 38 = expected points after 38 matches
Mark said: I accept the statistical noise for less than 2 points. I was just trying to look at how I can use a combination of tactics through the season to my advantage. Expand
One important thing to note, we are talking about 1-3 points RNG when there's 1,200 matches and we are talking about 0.5 - 1.5 points RNG when there's 4,000 matches.
But if we take 2,400 matches then in this case a tactic is tested against a specific formation only for 480 matches.... could you imagine the level of the RNG at such small distance.
For example, look at your result for "424 deformation II" tactic vs "5-2-3" AI formation, it's "42.5" points after 6 runs, but after 10 runs it drops to "36" points so the RNG of 6 runs is about "6.5" points.
As I said when the RNG could be as high as "6.5" points, an attempt to rank tactics in 0.5-2.5 points range doesn't make sense.
dzek said: How did you test all the formations? AI against AI? Expand
dzek said: If we had a H2H tactic testing league with all the existing formations that the game has for AI managers then we would have a more accurate answer I believe on this case. Expand
Hi,
If you take the best 200 AI managers in the game then:
- 35% of them have 4-3-3 Preferred Formation - 35% of them have 4-2-3-1 Preferred Formation - 20% of them have "different variations of 5 at the back formation" Preferred Formation - 10% of them have 4-4-2 Preferred Formation
So 70% of 200 best AI managers in the game play with 4-3-3 or 4-2-3-1
Also, look at the performance of AI's 5-2-3, it's significantly worse than AI's 4-2-3-1 or AI's 4-2-3-1.
We tested the rest of AI formations and they did even worse than AI's 5-2-3 so what's the point in having in the testing league?
According to our data If one tactic does better than other tactic vs AI's 4-2-3-1 or AI's 4-2-3-1 then it also will do better against other AI formations which are even weaker.
So when the overall difference between tactics is just 1-2 points then when it comes to a specific formation then the difference is also 1-2 points and such small difference is just "statistical noise".
Because I've checked our DB and here's what I got:
Red General 4231 v1.0
424 CF Cabuloso
433 Tsukyomi I
According to your table, "Red General 4231 v1.0" tactic is ranked "8th" and "424 CF Cabuloso" tactic is ranked "1st" when it come to 4-3-3 formation but according to our numbers the difference is about 1.5 points which could be just "statistical noise".
If you ask me then I think it makes sense to "rank" tactics only when there's a meaningful difference between them, I'd say it should be at least 10-15 points difference but when the difference is just 1-2 points then it can be easily "statistical noise".
Also, according to your table "433 Tsukyomi I" tactic is ranked "1st" when it comes to 5-3-2 formation but that's not true.
Here're some conclusions that we can draw from the result above:
- obviously, "meta" attributes is the real thing
- it doesn't seem there're minimal/optimal levels for other non-meta attributes that must be reached
- "non-meta" attributes still also matter because increasing 4 meta attributes to 20 gave less boost to the score than increased only 1 meta attribute but having other "non-meta" attributes at a much higher level - https://fm-arena.com/tactic/7528-5-points-to-pace-for-all-positions/. Yes, "non-meta" attributes also matter but their impact is much less than the impact of "meta" attributes.
Also, there's one important thing to understand when in our tactic testing league we set Acceleration attribute of a player to "20" then his Accelerating attribute will be 5-6 points higher than the Acceleration attribute any other player in the league.
But when you play the game in normal game and you try to outsmart the AI mangers by focusing on the meta attributes, you really shouldn't expect "miracle results", especially, when you play in the top leagues such as English Premier League and other because just look at the players in the top clubs of such leagues:
As you can see there's a lot of players in the top clubs that have the meta attributes at 17-20 level so when you play in the top leagues and even if you manage to bring in your club players that have the meta attributes at 17-18 level then still you won't be dominate the league because there're many clubs that also have a lot of player with maximized meta attributes.
letsgo9 said: @Zippo@Droid Would make any difference test all the Hall of Fame tactics till 4000 matches? Expand
Hi,
On average, the change in score between 2,400 matches and 4,000 matches is just -/+ 1 point.
Bear in mind, it would take quite a lot computing power to test every tactic that gets in HoF for 4,000 matches and what for? Just to correct its score by -/+ 1 point?
Taking into consideration, the quite low attention that HoF tactics get in general, I'd say the idea to spend a low of computing power on testing them for 4,000 matches to correct their score just -/+ 1 point isn't a good one.
The score change:
1,200 matches -> 2,400 matches = -/+ 1-3 points
2,400 matches -> 4,000 matches = -/+ 0-1 points
So if a tactic got "58" points after 2,400 matches then it almost has no chances to get higher than "59" if it gets tested up to 4,000 matches.
I just want to let you know that we've decided if after 1200 matches a tactic gets a score that is 2 points away from the top tactic then it'll get an additional testing up to 2400 matches.
At the moment there're about 12 tactics that scored at least 61 points(2 points away from the top) after 1200 matches in the queue to get tested up to 2400 matches.
pixar said: wow. Sharp decline after the storm. I think you'll need to change the amount of matches made in tests. A 5 point drop is a serious amount.
I think the amount of 4000 matches in each test may tire the system. I'm not too sure about that, but it can be considered to be at least 3000-2500 matches.
Nope, pal. I'm afraid it won't work that way.
You have to check every of your tested tactic and make sure all of them have proper screenshots. If it's needed you have to run tests again to add proper screenshots.
Only when you can confirm that all your tested tactics and non-tested tactics have proper screenshots we start test your new tactics again.
Let me know when you're done.
Cheers.
Tactical screenshots have been updated
@CSTG KANE, can you confirm that now all your tested tactics and non-tested tactics have proper screenshots?
An example:
- The manager with 'Use of Subs' value 1 may not make subs in-match or may make them quite late - for example in the 85th minute.
- The manager with 'Use of Subs' value 20 to make subs very quickly and in every game.
This again would have an effect on the results as a more rested player in the game would have their ability at full capacity while those who are tired would be -3/-5 in every attribute due to fatigue.
Only the game devs know for sure how "Use of Subs" attribute works, we can only guess about it.
Anyway, regardless of its working mechanic, it would be the same for every tactic tested so every tactic tested is effected by it at the same degree and we're comparing tactics here.
Hi,
Yes, the AI managers rotate their teams but the alternative players for every position have the same attributes, morale and conditions as the main players, the alternative players are just copies of the main players so it can be considered as "no rotations happens"
dzek said: 2. Within matches the AI managers and assistant managers (of user controlled teams) do players subs but the 'Use of Subs' attribute is different for each of them individually. Does this play a role in the sub strategy and in the results and if so, does this also affect the RNG?
Thank you!
Subs must happen and restricting them would have unpredictable consequences for the reliability of the results.
Hi,
Unfortunately, I don't think something like that would be possible.
Cheers.
@A Smile, @dzek, @letsgo9, thank you for your opinions and involvement in this situation, it's much appreciated.
@CSTG KANE, I see only one fair solution to this situation: you have to fix all the threads that have the same screenshots and add correct ones. Until then non of your tactics will be tested. Let me know when you're done.
Cheers.
Hi,
We dropped that feature because the demand for game saves was low and the cost of hosting them was too high.
But we still keep all the game saves in an offline storage and I can provide them by a request.
There seems to be some obvious changes from last year, such as high LOE + Low def line being the default option last year, but now everything is much higher + much higher.
Sorry if it's been asked before.
The last year TIs testing was more like an experiment and during the process it became clear that the way TIs work depends on formation/roles/duties/mentality and so on... other words, if something works well in one tactic then it isn't necessary also work well in other tactics.
The above means that if you want to test the TIs in a proper way then you have to test all possible combinations of TIs/Mentality/Formations/Roles/Duties. That would be something unrealizable to do. Also, add to it retesting with patches.
So, guys... no, TIs testing won't happen.
The thing is, the order in which the result screenshots are displayed isn't the order in which the tests happened. The result screenshot are randomly shuffled.
Mark said: I also only looked at the first 6 tests (ie 2400 matches) for each HOF tactic to ensure they were also all comparable.
As you can see we've got 1,200 matches, 2,400 matches and 4,000 matches in our test result tables and we compare all them just fine.
You just need to calculate P.P.M.(Points Per Match) and then translate it for 38 matches.
Total Points / Total Matches Played = P.P.M.(Points Per Match)
P.P.M.(Points Per Match) x 38 = expected points after 38 matches
Mark said: I accept the statistical noise for less than 2 points. I was just trying to look at how I can use a combination of tactics through the season to my advantage.
One important thing to note, we are talking about 1-3 points RNG when there's 1,200 matches and we are talking about 0.5 - 1.5 points RNG when there's 4,000 matches.
But if we take 2,400 matches then in this case a tactic is tested against a specific formation only for 480 matches.... could you imagine the level of the RNG at such small distance.
For example, look at your result for "424 deformation II" tactic vs "5-2-3" AI formation, it's "42.5" points after 6 runs, but after 10 runs it drops to "36" points so the RNG of 6 runs is about "6.5" points.
As I said when the RNG could be as high as "6.5" points, an attempt to rank tactics in 0.5-2.5 points range doesn't make sense.
dzek said: How did you test all the formations?
AI against AI?
We use a much more sophisticate method for that.
Hi,
If you take the best 200 AI managers in the game then:
- 35% of them have 4-3-3 Preferred Formation
- 35% of them have 4-2-3-1 Preferred Formation
- 20% of them have "different variations of 5 at the back formation" Preferred Formation
- 10% of them have 4-4-2 Preferred Formation
So 70% of 200 best AI managers in the game play with 4-3-3 or 4-2-3-1
Also, look at the performance of AI's 5-2-3, it's significantly worse than AI's 4-2-3-1 or AI's 4-2-3-1.
We tested the rest of AI formations and they did even worse than AI's 5-2-3 so what's the point in having in the testing league?
According to our data If one tactic does better than other tactic vs AI's 4-2-3-1 or AI's 4-2-3-1 then it also will do better against other AI formations which are even weaker.
Best Of Both Worlds v4
424 deformation II
So when the overall difference between tactics is just 1-2 points then when it comes to a specific formation then the difference is also 1-2 points and such small difference is just "statistical noise".
Hi,
I have no idea what testing methodology was used to get those numbers so I can't say anything about it.
Xeno94 said: or the weighting on the editor highly value "decisions" for multiple positions.
If your are talking abut the official attribute weighting then it's been proven many times by our tests that they are far away from being "correct".
Are you sure your numbers are correct?
Because I've checked our DB and here's what I got:
Red General 4231 v1.0
424 CF Cabuloso
433 Tsukyomi I
According to your table, "Red General 4231 v1.0" tactic is ranked "8th" and "424 CF Cabuloso" tactic is ranked "1st" when it come to 4-3-3 formation but according to our numbers the difference is about 1.5 points which could be just "statistical noise".
If you ask me then I think it makes sense to "rank" tactics only when there's a meaningful difference between them, I'd say it should be at least 10-15 points difference but when the difference is just 1-2 points then it can be easily "statistical noise".
Also, according to your table "433 Tsukyomi I" tactic is ranked "1st" when it comes to 5-3-2 formation but that's not true.
Just wanted to wish you, your families and the team a very merry Christmas and a happy new year.
Thank you for all the hard work you've put in this year delivering outstanding testing
All the best
Chris
Thank you, @CBP87.
I wishing everyone hope, peace and lots of Christmas cookies this holiday season!
I'm sure many of you guys have seen the results of our Player Attributes Testing for FM24 - https://fm-arena.com/table/26-player-attributes-testing/
The results of the testing have raised many important questions and there're a few of them:
- Should you pay attention only to meta attributes such as Pace, Acceleration and some other?
- What about other attributes? Do they matter at all?
- Are there any "optimal" levels for non-meta attributes that must be reach before you can ignore them?
So we've done another testing in the hope that it can answers the questions above.
On the screenshots below you can find the default attributes of the players in our tactic testing league.
FORWARDS
MIDFIELDERS
CENTRAL DEFENDERS
FULL BACKS
Ok, now let's edit every player in the human controlled teams and set their non-meta attributes to "7" and their meta attributes to "20".
So the attributes of every player in the human controlled teams look like this:
IMPORTANT NOTICE: The CA of every player has decreased by 40-50 points.
Now, let's test re-test "Red General 4231 v1.0" tactic with the adjusted players - https://fm-arena.com/thread/8026-red-general-4231-v1-0/
Here're some conclusions that we can draw from the result above:
- obviously, "meta" attributes is the real thing
- it doesn't seem there're minimal/optimal levels for other non-meta attributes that must be reached
- "non-meta" attributes still also matter because increasing 4 meta attributes to 20 gave less boost to the score than increased only 1 meta attribute but having other "non-meta" attributes at a much higher level - https://fm-arena.com/tactic/7528-5-points-to-pace-for-all-positions/. Yes, "non-meta" attributes also matter but their impact is much less than the impact of "meta" attributes.
Also, there's one important thing to understand when in our tactic testing league we set Acceleration attribute of a player to "20" then his Accelerating attribute will be 5-6 points higher than the Acceleration attribute any other player in the league.
But when you play the game in normal game and you try to outsmart the AI mangers by focusing on the meta attributes, you really shouldn't expect "miracle results", especially, when you play in the top leagues such as English Premier League and other because just look at the players in the top clubs of such leagues:
As you can see there's a lot of players in the top clubs that have the meta attributes at 17-20 level so when you play in the top leagues and even if you manage to bring in your club players that have the meta attributes at 17-18 level then still you won't be dominate the league because there're many clubs that also have a lot of player with maximized meta attributes.
Hi,
On average, the change in score between 2,400 matches and 4,000 matches is just -/+ 1 point.
Bear in mind, it would take quite a lot computing power to test every tactic that gets in HoF for 4,000 matches and what for? Just to correct its score by -/+ 1 point?
Taking into consideration, the quite low attention that HoF tactics get in general, I'd say the idea to spend a low of computing power on testing them for 4,000 matches to correct their score just -/+ 1 point isn't a good one.
The score change:
1,200 matches -> 2,400 matches = -/+ 1-3 points
2,400 matches -> 4,000 matches = -/+ 0-1 points
So if a tactic got "58" points after 2,400 matches then it almost has no chances to get higher than "59" if it gets tested up to 4,000 matches.
I just want to let you know that we've decided if after 1200 matches a tactic gets a score that is 2 points away from the top tactic then it'll get an additional testing up to 2400 matches.
At the moment there're about 12 tactics that scored at least 61 points(2 points away from the top) after 1200 matches in the queue to get tested up to 2400 matches.
Cheers.
Hi,
As you might noticed every tactic has a star rating: "Excellent", "Good", "Ok" and so on...
If some tactics have the same rating then it means that our tactic testing considers them to be equally good in real game.
So you can pick any tactic you like with "Excellent" rating and you really won't notice any difference in real game.
Also, tactics with "Good" rating can be almost as good as "Excellent" tactic.
I think the amount of 4000 matches in each test may tire the system. I'm not too sure about that, but it can be considered to be at least 3000-2500 matches.
@Zippo
Here's our testing algorithm for the current patch(24.2.0):
- the top tactic gets 4,000 matches
- all tactics in Hall Of Fame get 2,400 matches
- all other tactics get 1,200 matches
Would be nice just to understand the baseline everything else is working off of. Thanks
FORWARDS
MIDFIELDERS
CENTRAL DEFENDERS
FULL BACKS