You could start a new game and advance through pre-season to the first league game and save. All the teams in the league should have fresh players with good tactical familiarity. Open the app and, one by one, create a fixture against each team by swapping out your opponent, calibrating attendances and saving. Then write a script to simulate 1000 matches for each fixture and aggregate the results.
I have probably made an error in using the harness to test Maeda & Osimhen with different teams.
I think what has happened is that I've switched the team, not realizing it resets the player list. So those findings where Maeda does better against certain teams compared to Osimhen would be erroneous.
Maeda:
Tottenham - W 51 D 24 L 25 1.65-2.30 Arsenal - W 71 D 19 L 10 1.10-2.83 Aston Villa - W 65 D 18 L 17 1.18-2.70 Bournemouth - W 75 D 19 L 6 0.86-2.96 Brentford - W 75 D 16 L 9 1.05-3.02 Brighton - W 85 D 8 L 7 1.09-3.73 Burnley - W 89 D 7 L 4 0.65-3.57 Chelsea - W 84 D 9 L 7 0.98-3.46 Liverpool - W 27 D 18 L 55 2.12-1.75 Man City - W 62 D 15 L 23 1.30-2.30 Man UFC - W 82 D 14 L 4 1.05-3.11
Average Win rate = 69.64%
Osimhen:
Tottenham - W 56 D 14 L 30 1.79-2.93 Arsenal - W 81 D 13 L 6 1.07-3.75 Aston Villa - W 87 D 6 L 7 1.06-3.60 Bournemouth - W 89 D 10 L 1 0.88-4.41 Brentford - W 90 D 7 L 3 0.86-4.26 Brighton - W 88 D 8 L 4 1.09-5.12 Burnley - W 100 D 0 L 0 0.52-6.45 Chelsea - W 92 D 7 L 0 0.94-4.82 Liverpool - W 54 D 20 L 26 1.89-2.68 Man City - W 61 D 22 L 17 1.20-2.52 Man UFC - W 91 D 6 L 3 0.96-4.84
Average Win rate = 80.82%
Osimhen really is consistently better than Maeda using the harness.
So the question is how did Maeda do better in manual testing. It's unlikely to be due to random variation, as I took 7 samples of Maeda and 8 of Osimhen.
It occurred to me that perhaps it's because in the manual test, I had identical backups for the players, so perhaps it's the substitutions where the difference is made.
For instance, it could have been that because Maeda has significantly higher stamina and work rate than Osimhen, he was able to play on without being substituted, whereas Osimhen would be substituted for another Osimhen who has only ~50% match fitness due to rarely being played, which we know has a big impact on performance.
If true, it would mean that certain attributes such as stamina and perhaps work rate are overestimated in the derived attribute weights.
But also if that is the case, then there's still some truth in it, because in realistic play, having low match fitness subs is typical. So from that perspective, you could say maybe stamina & work rate do actually do deserve the extra weighting overall. You could probably add attributes such as determination and concentration to that.
I went back and looked at the old saves to see if this substitution theory holds up:
Osimhen:
34 (6) 32 (5)
44 (1) 40 (1)
44 (2) 43 (1)
47 (1) 39 (10)
44 (1) 43 (0)
Maeda:
50 (2) 47 (3)
49 (3) 45 (6)
48 (0) 47 (1)
55 51 (5)
55 (2) 54 (4)
56 (4) 52 (1)
Also not shown here, but observed, is that there was usually a 3rd Osimhim with a high number of starts, whereas for Maeda it was always just the main 2.
This is convincing enough for me to believe the substitution theory I've come up with.
So overall I think the new harness results are the most accurate, and we can use them for player rankings, but the weightings should probably still be adjusted to account for the reality that a player that can last out a match (on account of his stamina, work rate, concentration, etc.) is going to be better in practical use than a player that is a bit better on paper but can't last the match.
Apologies for going all over the place these last few posts, but sometimes that's how it goes.
( edited 7 hours, 3 min ago by GeorgeFloydOverdosed )
Share this post#903
Link to the post:
I further tested the old saves to verify:
Maeda (2 fit) - W 20 D 20 L 60 Osimhen (2 fit) - W 26 D 20 L 54 Osimhen (1 fit) - W 25 D 20 L 55 Maeda (1 fit) - W 17 D 25 L 58
So fitness doesn't make the difference.. Osimhen still does better. Yet in these same very saves, the Maeda team always does equal or better at the end of the season.
So I took a look at some of the losses in an Osimhen save to investigate what could be causing it.
It turns out that all 4 Osimhen players got called up for the Nigeria squad, leaving one ST spot to be filled by a greyed out player with attributes of ~7 and the other by an out-of-position meta player.
It only accounts for a few games, some of which were won anyway, but it's enough to make the difference. In the case I'm looking at, 2 games were probably lost due to these international call-ups.
The Japan team uses the EDT file which greys it out, so Maeda & Furuhashi benefited from never having international call-ups.
I suppose this is why apart from Furuhashi, Maeda and perhaps one or two others, the rankings are still mostly consistent - because everyone else gets called up for internationals as well.
In a way it's kind of realistic, because international call-ups do happen and therefore effect what you get out of a player, but that player shouldn't be replaced by a greyed out player, it should be a player only a bit less good.
I think this explains the discrepancy now. It's something I should have thought about in my testing, but it never occurred to me.
I will probably begin redoing the weights based on the harness data soon.
Edit: I should add this doesn't affect my previous meta team attribute testing at all. For the meta team, I did make sure to exempt them from international duty. I just forgot about it for the real player ranking tests.
Edit2: Thinking about it some more.. it still doesn't fully explain why Maeda got better results than Osimhen. It accounts for at most a ~5% difference in both directions. Maybe its partly the substitution, partly the international call-ups, and perhaps some other unknown factor. This all also doesn't I think explain the altered Knap tactic discrepancy between the harness & the real full season result.
There is a feature inside the pre-game editor called "Test Competitions", in which you can simulate competitions instantly and check results in the editor itself. You can then make changes to the team and simulate again. You can make it simulate for several seasons at once, or simulate custom competitions with more matches or even just two teams playing many times over.
Just saying that, for the things you are testing, you could also use the tools that already exist.
LightningFlik said: I'm currently running an experiment in which I take the top 32 goalkeepers and pit them against each other in a league/knockout to see who is the best in the game.
[...]
David De Gea vs Yann Sommer: 406-216-378
Well I don't think anybody saw this coming.
Were there flaws in this test? For sure. I should have maximised each players' familiarity with the tactics. I shouldn't have used an overpowered corner tactic. I should probably spend 64,000 matches on every goalie playing each other rather than a knockout tournament format, but we live and learn.
Still though. David de Gea. And it wasn't even close. Expand
I know 'I don't think anybody saw this coming' is bait, but I'll swallow it. The only thing I'm surprised about in the outcome is that Yann Sommer barely made it to the final round. David De Gea has 19 Reflexes and 17 Agility, the only two attributes you should care about for GK based on the testing on the very website you are posting on. Other GK tests pointed to the same; I got similar results in a 5,000-game-per-attribute sample, but in addition, other attributes, especially acceleration and pace, showed limited importance.
Of course, forums have a life of their own, and people still believe Aerial Reach to be super-important or put ~11 Reflexes in their scouting templates. Meanwhile, anyone with a few hundred hours played knows how easy it is to train a goalkeeper in Agility, but how hard it is to increase Reflexes. Factor in the aspect of GK Reactions individual training, and it leads to a massive waste of PA compared to Agility and Balance. Even though my own testing gave Agility a slight lead over Reflexes, I will never accept a young GK with < 14 Reflexes, just because I won't be able to increase it much later.
For me, GK selection goes this way; of course, first he must want to join your team: 1. Young enough with high PA. 2. Reflexes 14-15+ 3. Agility 13+ 4. Appropriate Consistency, Important Matches, Determination, and Professionalism. 5. The PA-waste stats as low as possible: Decisions, Positioning, Bravery, Kicking, etc. They will grow later anyway. 6. Relevant Pace and Acceleration.
ZaZ said: There is a feature inside the pre-game editor called "Test Competitions", in which you can simulate competitions instantly and check results in the editor itself. You can then make changes to the team and simulate again. You can make it simulate for several seasons at once, or simulate custom competitions with more matches or even just two teams playing many times over.
Just saying that, for the things you are testing, you could also use the tools that already exist. Expand
But it's not the actual in-game match-engine, right? It's just something that quickly sims a match based on reputation and probabilities just to fill out results so you can see how your custom competition might shake out?
Admittedly I've never used this and I'm happy to be wrong but I'd be floored if there was a separate copy of the FM Match Engine inside the pre-game editor.
Taisho said: I know 'I don't think anybody saw this coming' is bait, but I'll swallow it. Expand
It genuinely wasn't. I was surprised because I think he barely made the cut when I looked at the top goalkeepers in the game and cut off the list at 32. You're right about those attributes being the ones everybody talks about but I don't see people in a scramble to sign him in games.
ZaZ said: There is a feature inside the pre-game editor called "Test Competitions", in which you can simulate competitions instantly and check results in the editor itself. You can then make changes to the team and simulate again. You can make it simulate for several seasons at once, or simulate custom competitions with more matches or even just two teams playing many times over.
Just saying that, for the things you are testing, you could also use the tools that already exist. Expand Interesting. I didn't know there was this feature in the pre-game editor. I assumed you could only do validation of competitions, without showing results.
I see why I never came across it, it's quite labyrinthine to get to.
My first thought is that it might be using the QME and you can't simulate using Knap tactic selected, but I suppose that the player rankings should still be about the same no matter what the tactic.
Even in regards to the QME, it processes very fast, but so does the harness, as it's skipping all the AI transfers and whatnot. It's just not clear whether its QME or full.
Seems useful overall. Probably wish I knew about this when I was doing my manual player rank testing.
GeorgeFloydOverdosed said: So what to do now..
You could start a new game and advance through pre-season to the first league game and save. All the teams in the league should have fresh players with good tactical familiarity. Open the app and, one by one, create a fixture against each team by swapping out your opponent, calibrating attendances and saving. Then write a script to simulate 1000 matches for each fixture and aggregate the results.
I have probably made an error in using the harness to test Maeda & Osimhen with different teams.
I think what has happened is that I've switched the team, not realizing it resets the player list. So those findings where Maeda does better against certain teams compared to Osimhen would be erroneous.
Maeda:
Tottenham - W 51 D 24 L 25 1.65-2.30
Arsenal - W 71 D 19 L 10 1.10-2.83
Aston Villa - W 65 D 18 L 17 1.18-2.70
Bournemouth - W 75 D 19 L 6 0.86-2.96
Brentford - W 75 D 16 L 9 1.05-3.02
Brighton - W 85 D 8 L 7 1.09-3.73
Burnley - W 89 D 7 L 4 0.65-3.57
Chelsea - W 84 D 9 L 7 0.98-3.46
Liverpool - W 27 D 18 L 55 2.12-1.75
Man City - W 62 D 15 L 23 1.30-2.30
Man UFC - W 82 D 14 L 4 1.05-3.11
Average Win rate = 69.64%
Osimhen:
Tottenham - W 56 D 14 L 30 1.79-2.93
Arsenal - W 81 D 13 L 6 1.07-3.75
Aston Villa - W 87 D 6 L 7 1.06-3.60
Bournemouth - W 89 D 10 L 1 0.88-4.41
Brentford - W 90 D 7 L 3 0.86-4.26
Brighton - W 88 D 8 L 4 1.09-5.12
Burnley - W 100 D 0 L 0 0.52-6.45
Chelsea - W 92 D 7 L 0 0.94-4.82
Liverpool - W 54 D 20 L 26 1.89-2.68
Man City - W 61 D 22 L 17 1.20-2.52
Man UFC - W 91 D 6 L 3 0.96-4.84
Average Win rate = 80.82%
Osimhen really is consistently better than Maeda using the harness.
So the question is how did Maeda do better in manual testing. It's unlikely to be due to random variation, as I took 7 samples of Maeda and 8 of Osimhen.
It occurred to me that perhaps it's because in the manual test, I had identical backups for the players, so perhaps it's the substitutions where the difference is made.
For instance, it could have been that because Maeda has significantly higher stamina and work rate than Osimhen, he was able to play on without being substituted, whereas Osimhen would be substituted for another Osimhen who has only ~50% match fitness due to rarely being played, which we know has a big impact on performance.
If true, it would mean that certain attributes such as stamina and perhaps work rate are overestimated in the derived attribute weights.
But also if that is the case, then there's still some truth in it, because in realistic play, having low match fitness subs is typical. So from that perspective, you could say maybe stamina & work rate do actually do deserve the extra weighting overall. You could probably add attributes such as determination and concentration to that.
I went back and looked at the old saves to see if this substitution theory holds up:
Osimhen:
34 (6)
32 (5)
44 (1)
40 (1)
44 (2)
43 (1)
47 (1)
39 (10)
44 (1)
43 (0)
Maeda:
50 (2)
47 (3)
49 (3)
45 (6)
48 (0)
47 (1)
55
51 (5)
55 (2)
54 (4)
56 (4)
52 (1)
Also not shown here, but observed, is that there was usually a 3rd Osimhim with a high number of starts, whereas for Maeda it was always just the main 2.
This is convincing enough for me to believe the substitution theory I've come up with.
So overall I think the new harness results are the most accurate, and we can use them for player rankings, but the weightings should probably still be adjusted to account for the reality that a player that can last out a match (on account of his stamina, work rate, concentration, etc.) is going to be better in practical use than a player that is a bit better on paper but can't last the match.
Apologies for going all over the place these last few posts, but sometimes that's how it goes.
I further tested the old saves to verify:
Maeda (2 fit) - W 20 D 20 L 60
Osimhen (2 fit) - W 26 D 20 L 54
Osimhen (1 fit) - W 25 D 20 L 55
Maeda (1 fit) - W 17 D 25 L 58
So fitness doesn't make the difference.. Osimhen still does better. Yet in these same very saves, the Maeda team always does equal or better at the end of the season.
So I took a look at some of the losses in an Osimhen save to investigate what could be causing it.
It turns out that all 4 Osimhen players got called up for the Nigeria squad, leaving one ST spot to be filled by a greyed out player with attributes of ~7 and the other by an out-of-position meta player.
It only accounts for a few games, some of which were won anyway, but it's enough to make the difference. In the case I'm looking at, 2 games were probably lost due to these international call-ups.
The Japan team uses the EDT file which greys it out, so Maeda & Furuhashi benefited from never having international call-ups.
I suppose this is why apart from Furuhashi, Maeda and perhaps one or two others, the rankings are still mostly consistent - because everyone else gets called up for internationals as well.
In a way it's kind of realistic, because international call-ups do happen and therefore effect what you get out of a player, but that player shouldn't be replaced by a greyed out player, it should be a player only a bit less good.
I think this explains the discrepancy now. It's something I should have thought about in my testing, but it never occurred to me.
I will probably begin redoing the weights based on the harness data soon.
Edit: I should add this doesn't affect my previous meta team attribute testing at all. For the meta team, I did make sure to exempt them from international duty. I just forgot about it for the real player ranking tests.
Edit2: Thinking about it some more.. it still doesn't fully explain why Maeda got better results than Osimhen. It accounts for at most a ~5% difference in both directions. Maybe its partly the substitution, partly the international call-ups, and perhaps some other unknown factor. This all also doesn't I think explain the altered Knap tactic discrepancy between the harness & the real full season result.
There is a feature inside the pre-game editor called "Test Competitions", in which you can simulate competitions instantly and check results in the editor itself. You can then make changes to the team and simulate again. You can make it simulate for several seasons at once, or simulate custom competitions with more matches or even just two teams playing many times over.
Just saying that, for the things you are testing, you could also use the tools that already exist.
LightningFlik said: I'm currently running an experiment in which I take the top 32 goalkeepers and pit them against each other in a league/knockout to see who is the best in the game.
[...]
David De Gea vs Yann Sommer: 406-216-378
Well I don't think anybody saw this coming.
Were there flaws in this test? For sure. I should have maximised each players' familiarity with the tactics. I shouldn't have used an overpowered corner tactic. I should probably spend 64,000 matches on every goalie playing each other rather than a knockout tournament format, but we live and learn.
Still though. David de Gea. And it wasn't even close.
I know 'I don't think anybody saw this coming' is bait, but I'll swallow it. The only thing I'm surprised about in the outcome is that Yann Sommer barely made it to the final round. David De Gea has 19 Reflexes and 17 Agility, the only two attributes you should care about for GK based on the testing on the very website you are posting on. Other GK tests pointed to the same; I got similar results in a 5,000-game-per-attribute sample, but in addition, other attributes, especially acceleration and pace, showed limited importance.
Of course, forums have a life of their own, and people still believe Aerial Reach to be super-important or put ~11 Reflexes in their scouting templates. Meanwhile, anyone with a few hundred hours played knows how easy it is to train a goalkeeper in Agility, but how hard it is to increase Reflexes. Factor in the aspect of GK Reactions individual training, and it leads to a massive waste of PA compared to Agility and Balance. Even though my own testing gave Agility a slight lead over Reflexes, I will never accept a young GK with < 14 Reflexes, just because I won't be able to increase it much later.
For me, GK selection goes this way; of course, first he must want to join your team:
1. Young enough with high PA.
2. Reflexes 14-15+
3. Agility 13+
4. Appropriate Consistency, Important Matches, Determination, and Professionalism.
5. The PA-waste stats as low as possible: Decisions, Positioning, Bravery, Kicking, etc. They will grow later anyway.
6. Relevant Pace and Acceleration.
ZaZ said: There is a feature inside the pre-game editor called "Test Competitions", in which you can simulate competitions instantly and check results in the editor itself. You can then make changes to the team and simulate again. You can make it simulate for several seasons at once, or simulate custom competitions with more matches or even just two teams playing many times over.
Just saying that, for the things you are testing, you could also use the tools that already exist.
But it's not the actual in-game match-engine, right? It's just something that quickly sims a match based on reputation and probabilities just to fill out results so you can see how your custom competition might shake out?
Admittedly I've never used this and I'm happy to be wrong but I'd be floored if there was a separate copy of the FM Match Engine inside the pre-game editor.
Taisho said: I know 'I don't think anybody saw this coming' is bait, but I'll swallow it.
It genuinely wasn't. I was surprised because I think he barely made the cut when I looked at the top goalkeepers in the game and cut off the list at 32. You're right about those attributes being the ones everybody talks about but I don't see people in a scramble to sign him in games.
ZaZ said: There is a feature inside the pre-game editor called "Test Competitions", in which you can simulate competitions instantly and check results in the editor itself. You can then make changes to the team and simulate again. You can make it simulate for several seasons at once, or simulate custom competitions with more matches or even just two teams playing many times over.
Just saying that, for the things you are testing, you could also use the tools that already exist.
Interesting. I didn't know there was this feature in the pre-game editor. I assumed you could only do validation of competitions, without showing results.
I see why I never came across it, it's quite labyrinthine to get to.
My first thought is that it might be using the QME and you can't simulate using Knap tactic selected, but I suppose that the player rankings should still be about the same no matter what the tactic.
Even in regards to the QME, it processes very fast, but so does the harness, as it's skipping all the AI transfers and whatnot. It's just not clear whether its QME or full.
Seems useful overall. Probably wish I knew about this when I was doing my manual player rank testing.