You could start a new game and advance through pre-season to the first league game and save. All the teams in the league should have fresh players with good tactical familiarity. Open the app and, one by one, create a fixture against each team by swapping out your opponent, calibrating attendances and saving. Then write a script to simulate 1000 matches for each fixture and aggregate the results.
I have probably made an error in using the harness to test Maeda & Osimhen with different teams.
I think what has happened is that I've switched the team, not realizing it resets the player list. So those findings where Maeda does better against certain teams compared to Osimhen would be erroneous.
Maeda:
Tottenham - W 51 D 24 L 25 1.65-2.30 Arsenal - W 71 D 19 L 10 1.10-2.83 Aston Villa - W 65 D 18 L 17 1.18-2.70 Bournemouth - W 75 D 19 L 6 0.86-2.96 Brentford - W 75 D 16 L 9 1.05-3.02 Brighton - W 85 D 8 L 7 1.09-3.73 Burnley - W 89 D 7 L 4 0.65-3.57 Chelsea - W 84 D 9 L 7 0.98-3.46 Liverpool - W 27 D 18 L 55 2.12-1.75 Man City - W 62 D 15 L 23 1.30-2.30 Man UFC - W 82 D 14 L 4 1.05-3.11
Average Win rate = 69.64%
Osimhen:
Tottenham - W 56 D 14 L 30 1.79-2.93 Arsenal - W 81 D 13 L 6 1.07-3.75 Aston Villa - W 87 D 6 L 7 1.06-3.60 Bournemouth - W 89 D 10 L 1 0.88-4.41 Brentford - W 90 D 7 L 3 0.86-4.26 Brighton - W 88 D 8 L 4 1.09-5.12 Burnley - W 100 D 0 L 0 0.52-6.45 Chelsea - W 92 D 7 L 0 0.94-4.82 Liverpool - W 54 D 20 L 26 1.89-2.68 Man City - W 61 D 22 L 17 1.20-2.52 Man UFC - W 91 D 6 L 3 0.96-4.84
Average Win rate = 80.82%
Osimhen really is consistently better than Maeda using the harness.
So the question is how did Maeda do better in manual testing. It's unlikely to be due to random variation, as I took 7 samples of Maeda and 8 of Osimhen.
It occurred to me that perhaps it's because in the manual test, I had identical backups for the players, so perhaps it's the substitutions where the difference is made.
For instance, it could have been that because Maeda has significantly higher stamina and work rate than Osimhen, he was able to play on without being substituted, whereas Osimhen would be substituted for another Osimhen who has only ~50% match fitness due to rarely being played, which we know has a big impact on performance.
If true, it would mean that certain attributes such as stamina and perhaps work rate are overestimated in the derived attribute weights.
But also if that is the case, then there's still some truth in it, because in realistic play, having low match fitness subs is typical. So from that perspective, you could say maybe stamina & work rate do actually do deserve the extra weighting overall. You could probably add attributes such as determination and concentration to that.
I went back and looked at the old saves to see if this substitution theory holds up:
Osimhen:
34 (6) 32 (5)
44 (1) 40 (1)
44 (2) 43 (1)
47 (1) 39 (10)
44 (1) 43 (0)
Maeda:
50 (2) 47 (3)
49 (3) 45 (6)
48 (0) 47 (1)
55 51 (5)
55 (2) 54 (4)
56 (4) 52 (1)
Also not shown here, but observed, is that there was usually a 3rd Osimhim with a high number of starts, whereas for Maeda it was always just the main 2.
This is convincing enough for me to believe the substitution theory I've come up with.
So overall I think the new harness results are the most accurate, and we can use them for player rankings, but the weightings should probably still be adjusted to account for the reality that a player that can last out a match (on account of his stamina, work rate, concentration, etc.) is going to be better in practical use than a player that is a bit better on paper but can't last the match.
Apologies for going all over the place these last few posts, but sometimes that's how it goes.
( edited 2 hours, 53 min ago by GeorgeFloydOverdosed )
Share this post#903
Link to the post:
I further tested the old saves to verify:
Maeda (2 fit) - W 20 D 20 L 60 Osimhen (2 fit) - W 26 D 20 L 54 Osimhen (1 fit) - W 25 D 20 L 55 Maeda (1 fit) - W 17 D 25 L 58
So fitness doesn't make the difference.. Osimhen still does better. Yet in these same very saves, the Maeda team always does equal or better at the end of the season.
So I took a look at some of the losses in an Osimhen save to investigate what could be causing it.
It turns out that all 4 Osimhen players got called up for the Nigeria squad, leaving one ST spot to be filled by a greyed out player with attributes of ~7 and the other by an out-of-position meta player.
It only accounts for a few games, some of which were won anyway, but it's enough to make the difference. In the case I'm looking at, 2 games were probably lost due to these international call-ups.
The Japan team uses the EDT file which greys it out, so Maeda & Furuhashi benefited from never having international call-ups.
I suppose this is why apart from Furuhashi, Maeda and perhaps one or two others, the rankings are still mostly consistent - because everyone else gets called up for internationals as well.
In a way it's kind of realistic, because international call-ups do happen and therefore effect what you get out of a player, but that player shouldn't be replaced by a greyed out player, it should be a player only a bit less good.
I think this explains the discrepancy now. It's something I should have thought about in my testing, but it never occurred to me.
I will probably begin redoing the weights based on the harness data soon.
Edit: I should add this doesn't affect my previous meta team attribute testing at all. For the meta team, I did make sure to exempt them from international duty. I just forgot about it for the real player ranking tests.
Edit2: Thinking about it some more.. it still doesn't fully explain why Maeda got better results than Osimhen. It accounts for at most a ~5% difference in both directions. Maybe its partly the substitution, partly the international call-ups, and perhaps some other unknown factor. This all also doesn't I think explain the altered Knap tactic discrepancy between the harness & the real full season result.
There is a feature inside the pre-game editor called "Test Competitions", in which you can simulate competitions instantly and check results in the editor itself. You can then make changes to the team and simulate again. You can make it simulate for several seasons at once, or simulate custom competitions with more matches or even just two teams playing many times over.
Just saying that, for the things you are testing, you could also use the tools that already exist.
GeorgeFloydOverdosed said: So what to do now..
You could start a new game and advance through pre-season to the first league game and save. All the teams in the league should have fresh players with good tactical familiarity. Open the app and, one by one, create a fixture against each team by swapping out your opponent, calibrating attendances and saving. Then write a script to simulate 1000 matches for each fixture and aggregate the results.
I have probably made an error in using the harness to test Maeda & Osimhen with different teams.
I think what has happened is that I've switched the team, not realizing it resets the player list. So those findings where Maeda does better against certain teams compared to Osimhen would be erroneous.
Maeda:
Tottenham - W 51 D 24 L 25 1.65-2.30
Arsenal - W 71 D 19 L 10 1.10-2.83
Aston Villa - W 65 D 18 L 17 1.18-2.70
Bournemouth - W 75 D 19 L 6 0.86-2.96
Brentford - W 75 D 16 L 9 1.05-3.02
Brighton - W 85 D 8 L 7 1.09-3.73
Burnley - W 89 D 7 L 4 0.65-3.57
Chelsea - W 84 D 9 L 7 0.98-3.46
Liverpool - W 27 D 18 L 55 2.12-1.75
Man City - W 62 D 15 L 23 1.30-2.30
Man UFC - W 82 D 14 L 4 1.05-3.11
Average Win rate = 69.64%
Osimhen:
Tottenham - W 56 D 14 L 30 1.79-2.93
Arsenal - W 81 D 13 L 6 1.07-3.75
Aston Villa - W 87 D 6 L 7 1.06-3.60
Bournemouth - W 89 D 10 L 1 0.88-4.41
Brentford - W 90 D 7 L 3 0.86-4.26
Brighton - W 88 D 8 L 4 1.09-5.12
Burnley - W 100 D 0 L 0 0.52-6.45
Chelsea - W 92 D 7 L 0 0.94-4.82
Liverpool - W 54 D 20 L 26 1.89-2.68
Man City - W 61 D 22 L 17 1.20-2.52
Man UFC - W 91 D 6 L 3 0.96-4.84
Average Win rate = 80.82%
Osimhen really is consistently better than Maeda using the harness.
So the question is how did Maeda do better in manual testing. It's unlikely to be due to random variation, as I took 7 samples of Maeda and 8 of Osimhen.
It occurred to me that perhaps it's because in the manual test, I had identical backups for the players, so perhaps it's the substitutions where the difference is made.
For instance, it could have been that because Maeda has significantly higher stamina and work rate than Osimhen, he was able to play on without being substituted, whereas Osimhen would be substituted for another Osimhen who has only ~50% match fitness due to rarely being played, which we know has a big impact on performance.
If true, it would mean that certain attributes such as stamina and perhaps work rate are overestimated in the derived attribute weights.
But also if that is the case, then there's still some truth in it, because in realistic play, having low match fitness subs is typical. So from that perspective, you could say maybe stamina & work rate do actually do deserve the extra weighting overall. You could probably add attributes such as determination and concentration to that.
I went back and looked at the old saves to see if this substitution theory holds up:
Osimhen:
34 (6)
32 (5)
44 (1)
40 (1)
44 (2)
43 (1)
47 (1)
39 (10)
44 (1)
43 (0)
Maeda:
50 (2)
47 (3)
49 (3)
45 (6)
48 (0)
47 (1)
55
51 (5)
55 (2)
54 (4)
56 (4)
52 (1)
Also not shown here, but observed, is that there was usually a 3rd Osimhim with a high number of starts, whereas for Maeda it was always just the main 2.
This is convincing enough for me to believe the substitution theory I've come up with.
So overall I think the new harness results are the most accurate, and we can use them for player rankings, but the weightings should probably still be adjusted to account for the reality that a player that can last out a match (on account of his stamina, work rate, concentration, etc.) is going to be better in practical use than a player that is a bit better on paper but can't last the match.
Apologies for going all over the place these last few posts, but sometimes that's how it goes.
I further tested the old saves to verify:
Maeda (2 fit) - W 20 D 20 L 60
Osimhen (2 fit) - W 26 D 20 L 54
Osimhen (1 fit) - W 25 D 20 L 55
Maeda (1 fit) - W 17 D 25 L 58
So fitness doesn't make the difference.. Osimhen still does better. Yet in these same very saves, the Maeda team always does equal or better at the end of the season.
So I took a look at some of the losses in an Osimhen save to investigate what could be causing it.
It turns out that all 4 Osimhen players got called up for the Nigeria squad, leaving one ST spot to be filled by a greyed out player with attributes of ~7 and the other by an out-of-position meta player.
It only accounts for a few games, some of which were won anyway, but it's enough to make the difference. In the case I'm looking at, 2 games were probably lost due to these international call-ups.
The Japan team uses the EDT file which greys it out, so Maeda & Furuhashi benefited from never having international call-ups.
I suppose this is why apart from Furuhashi, Maeda and perhaps one or two others, the rankings are still mostly consistent - because everyone else gets called up for internationals as well.
In a way it's kind of realistic, because international call-ups do happen and therefore effect what you get out of a player, but that player shouldn't be replaced by a greyed out player, it should be a player only a bit less good.
I think this explains the discrepancy now. It's something I should have thought about in my testing, but it never occurred to me.
I will probably begin redoing the weights based on the harness data soon.
Edit: I should add this doesn't affect my previous meta team attribute testing at all. For the meta team, I did make sure to exempt them from international duty. I just forgot about it for the real player ranking tests.
Edit2: Thinking about it some more.. it still doesn't fully explain why Maeda got better results than Osimhen. It accounts for at most a ~5% difference in both directions. Maybe its partly the substitution, partly the international call-ups, and perhaps some other unknown factor. This all also doesn't I think explain the altered Knap tactic discrepancy between the harness & the real full season result.
There is a feature inside the pre-game editor called "Test Competitions", in which you can simulate competitions instantly and check results in the editor itself. You can then make changes to the team and simulate again. You can make it simulate for several seasons at once, or simulate custom competitions with more matches or even just two teams playing many times over.
Just saying that, for the things you are testing, you could also use the tools that already exist.