Welcome to symthic forums! We would love if you'd register!
You don't have to be expert in bit baking, everyone is more than welcome to join our community.

You are not logged in.

Quick Statistics - Does rank/gameplay time tell skill?

Hey! If this is your first visit on symthic.com, also check out our weapon damage charts.
Currently we have charts for Battlefield 3, Call of Duty: Black Ops 2, Medal of Honor: Warfighter and Call of Duty: Modern Warfare 3

Symthic Developer

Posts: 3,694

Date of registration
: Mar 21st 2013

Platform: PC

Location: __main__, Finland

Reputation modifier: 16

Monday, April 7th 2014, 2:21pm

Quick Statistics - Does rank/gameplay time tell skill?

Thanks to pmax (throw more rep at him, nauw) for providing us with nice and big sample of bf4 players I finally managed to bring my lazy butt to do this, which is about proving statement "Playtime/Rank doesn't tell how good player is" false or true via statistical means.
Do note thought: I am rather newb and new at all this even tho I take proper courses on statistics, so there might be horrible mistakes. Don't worry though, 3VN or pmax will hang me and notify if I managed to derp up completely.

Alright so, the problem here was that there's simply no mean to tell if player is good. At best we have values that might tell something (KD, SPM, KPM, etc), but alone none of those tells true skill. So our best bet is to compare gameplay time and/or rank to all these and see what we come up with.

Rank is bit more discrete variable than the others, but it should be okey for these analysis (It has zero point and you can compare them, plus there's +100 different values for this).

Correlations:

Source code

1
2
3
4
5
6
7
8
9
10
11
12

Variables |  Correlation & P
----------+--------------------
time+kd  |  0.135 (p <<< 0.01)
time+spm  |  0.266 (p <<< 0.01)
time+kpm  |  0.246 (p <<< 0.01)
rank+kd  |  0.244 (p <<< 0.01)
rank+spm  |  0.688 (p <<< 0.01)
rank+kpm  |  0.524 (p <<< 0.01)
|
kd+spm  |  0.336 (p <<< 0.01)
kd+kpm  |  0.371 (p <<< 0.01)
spm+kpm  |  0.709 (p <<< 0.01)

Quick info on correlation

High correlation (close to 1 or -1) signs about connection between the two variables in question. If it's close to 1, scatterplot should look similar to a rising line (when other of the variables rise, the other does as well). If it's close to -1, other variable will decrease as other rises. HOWEVER, high correlation doesn't tell about cause and effect, only that we can assume there's a connection between these two.
And about P-value: If P value is low (usually <0.05 or 0.01), then the analysis is statistically valid. As in we can assume randomness didn't have effect on the sample.

The correlations are surprisingly low for other thank rank+spm/kpm. However due to the size of the set we can safely assume there are some correlation between them. This can be also explained by the fact these don't measure true skill, only some parts of it at best. In plots (see end of the post) we can see slight patterns between variables, which of some seem to be slightly curving so correlation doesn't count those in too well.
In pure correlation values, rank+spm and rank+kpm stand out really well. However in the plots they don't seem to differ from rank+kd too much. There's a somewhat distinct line in rank+spm hex plot (heatmappish) where I used all of the samples. This might be due to missing spots in plots matrix picture (used every 100th sample). Rank+spm could be explained with usage of xp boosts and rank+kpm with how some play mostly gamemodes where you gain most of your score from kills. These are just assumptions tho.

As a extra I also compared these "skill" values together. KD compared to others has reasonable correlation which is also visible in the plots, there also seems to be quite a many outliers so this might lower the correlation quite a bit. SPM+KPM has surprisingly high correlation and plots show distinct rising line/block, this signs about how BF4 is still mostly kill based game even tho there are objectives and other ways to gain score than kills.

TL;DR / (My) Conclusion: While one can't say for sure if one is a good play just by looking at the rank, there most likely is some connection between them (playing the game -> learning -> getting better). Especially high rank could sign about high SPM and KPM. SPM and KPM are rather connected to eachother.
Update: By creating a product from SPM, KD and KPM and calculating correlation of that against rank and time with Spearman's correlation test we get 0.670 against rank and 0.4 against gameplaytime. This supports the original conclusion that rank/gameplay time indeed correlates with skill and thus you can assume player with high rank or gameplay time is more skillful than one with lower rank/gameplaytime.

Rank+spm hex plot

UPDATE: Stuff I posted in replies.

Quoted from "Miffyli"

Alright I am not too sure if there's a good or special method of creating nice product of all these, but I got correlation of 0.507 against rank and 0.236 against gameplay time when I calculated the variable like this:

Source code

1

kd/median(kd) + spm/median(spm) + kpm/median(kpm)

Source code

1
2
3
4
5
6
7
8
9
10
11
12

Variables |  Correlation (rho) & P
----------+--------------------
time+kd  |  0.367 (p <<< 0.01)
time+spm  |  0.427 (p <<< 0.01)
time+kpm  |  0.323 (p <<< 0.01)
rank+kd  |  0.567 (p <<< 0.01)
rank+spm  |  0.734 (p <<< 0.01)
rank+kpm  |  0.540 (p <<< 0.01)
|
kd+spm  |  0.704 (p <<< 0.01)
kd+kpm  |  0.831 (p <<< 0.01)
spm+kpm  |  0.718 (p <<< 0.01)

And with the product of skill variables (with *) I get 0.670 against rank and 0.405 against time.

Quoted

Alright, I calculated log(kd) and did correlation tests against that (This time I assume I should use person's, because log(kd) is almost normally distributed?). I get 0.547 against rank and 0.303 against gameplaytime.
As for doing product with summing and z values I get following (again, I assume pearson's is the one we should use because of normal distribution of product?):

Source code

1
2
3
4

Test      | vs. Rank | vs. Time
----------+----------+---------
Pearson's |    0.654 |   0.303
Spearman's|    0.664 |   0.396
• 3VerstsNorth - Analysis of game mechanics in BF4 (tickrates, effects of tickrate, etc)
• leptis - Analysis of shotguns, recoil, recoil control and air drag.
• Veritable - Scoring of BF4/BF1 firearms in terms of usability, firing and other mechanics.
• Miffyli - Random statistical analysis of BF4 battlereports/players and kill-distances. (list is cluttered with other threads).
Sorry if your name wasn't on the list, I honestly can't recall all names : ( . Nudge me if you want to be included

This post has been edited 7 times, last edit by "Miffyli" (Apr 10th 2014, 6:28pm)

Skynet T100

Posts: 1,614

Date of registration
: Apr 12th 2013

Platform: PC

Location: Guilin Peaks, Finland

Battlelog:

Reputation modifier: 14

Monday, April 7th 2014, 2:46pm

Quoted from "Miffyli"

"Playtime/Rank doesn't tell how good player is"

So, your analysis conclusively shows that this hypothesis has to be rejected because playtime and rank were both highly significantly correlated with the skill variables (KPM, SPM, and to a lesser degree KDR). That's a clearcut solid result, props.

As to the effect size, the correlations of rank with KPM and SPM were really quite strong, which indicates that this result is not just a statistical gimmick but rather shows a concrete phenomenon. Of course it is not unexpected that players learn to play the game while they are playing it, but this is a good foundation for addressing follow-up questions. E.g., look at the residual variability in the "skill" of max rank players.

Could you plot the distribution of ranks? The apparent lack of high-but-not-max-rank players speaks for a segregation of the playerbase to occasional/casual/drop-out and hardcore gamers.

You could also test using the product of KDR, KPM, and SPM as a skill metric to see if that gives a stronger correlation with the investment metric (time, rank).

As to confounders, time is of course confounded by the fact that prior experience is not accounted for. Rank on the other hand is essentially the product of time and SPM, and there not a variable independent of SPM (=> high correlation between dependent variables is like a tautology).

Anyway, great work!

Extra rep for the cool matrix visualization.
"Less is more? How can that be? How could less be more, that's impossible. More is more." Yngwie Malmsten
"Many bullets help." WoopsyYaya
"most rhetorically legitimate ad hominem 2015" ToTheSun!

Symthic Developer

Posts: 3,694

Date of registration
: Mar 21st 2013

Platform: PC

Location: __main__, Finland

Reputation modifier: 16

Monday, April 7th 2014, 2:56pm

Quoted from "3VerstsNorth"

Could you plot the distribution of ranks? The apparent lack of high-but-not-max-rank players speaks for a segregation of the playerbase to occasional/casual/drop-out and hardcore gamers.

histogram

And fivenum summary: 0 37 60 92 120

Almost normal, but that interval with 110 really breaks the model.
I guess this is where the method used to collect the sample comes in: Since these players are from forums of the game, it's expected there are more hardcoreish players there than casuals.
I'll check those other requests of yours after I have done some HW. Doing statistics for actual set was a lot more fun than semi-random HW sample

Quoted from "3VerstsNorth"

As to confounders, time is of course confounded by the fact that prior experience is not accounted for. Rank on the other hand is essentially the product of time and SPM, and there not a variable independent of SPM (=> high correlation between dependent variables is like a tautology).

Oh yes right! How did I not think this through... >.<
• 3VerstsNorth - Analysis of game mechanics in BF4 (tickrates, effects of tickrate, etc)
• leptis - Analysis of shotguns, recoil, recoil control and air drag.
• Veritable - Scoring of BF4/BF1 firearms in terms of usability, firing and other mechanics.
• Miffyli - Random statistical analysis of BF4 battlereports/players and kill-distances. (list is cluttered with other threads).
Sorry if your name wasn't on the list, I honestly can't recall all names : ( . Nudge me if you want to be included

Posts: 42

Date of registration
: Apr 4th 2014

Platform: PC

Battlelog:

Reputation modifier: 2

Monday, April 7th 2014, 3:25pm

Some people will never get good, regardless of how much they play. Some people are just naturally good at FPS. Some people will play enough hours that they get good. I don't think it's easy to say one way or the other.

For me, I just built my first PC for gaming in late December...had never gamed on a PC until then. Also, I had never played a Battlefield game. I spent A LOT of hours being pretty worthless, I struggled a lot early on. Now, around 550 hours later I have adjusted to the mouse and keyboard and the game. I find myself consistently at the top of the 64 player leaderboard pretty much every game. I would guess that if I started fresh as of about a month ago, my K/D is around 3 or maybe a bit lower (driven by me spending a lot of time in vehicles; as infantry I am def below a 2 because I play aggressive and try to cap constantly). I spent a lot of time mastering the Scout helicopter in particular, I have many 50+ kills and 10 or less death games -- while capping points in the air, I don't just farm kills, I try to get as many points for my team as possible.

Overall, if you look at my SPM, Skill, or K/D, K/min, they don't look great...they are heavily deflated by the first 300 hours or so where I was pretty damn bad. Now I would say I am pretty solid.

Posts: 250

Date of registration
: Jan 2nd 2014

Platform: PC

Battlelog:

Reputation modifier: 3

Monday, April 7th 2014, 4:09pm

Cool stuff!

What (if any) light does this shine on the acual in-game "Skill" stat. I know we've got the formula somewhere. From my experience, it tends to vary quite a bit. I imagine gamemode and similar factors have a fairly significant effect.

They must have had something in mind when programming it, surely? I just wonder if it actually shows the more skillful players (going from your analysis), or the ones who are slightly more consistent in their abilities or enjoy stomping TDM games with experienced clan teammates.
Roses are red, violets are blue, "They don't think it be like it is, but it do."

Symthic Developer

Posts: 3,694

Date of registration
: Mar 21st 2013

Platform: PC

Location: __main__, Finland

Reputation modifier: 16

Monday, April 7th 2014, 4:20pm

Quoted from "3VerstsNorth"

You could also test using the product of KDR, KPM, and SPM as a skill metric to see if that gives a stronger correlation with the investment metric (time, rank).

Alright I am not too sure if there's a good or special method of creating nice product of all these, but I got correlation of 0.507 against rank and 0.236 against gameplay time when I calculated the variable like this:

Source code

1

kd/median(kd) + spm/median(spm) + kpm/median(kpm)

Do you think this is "good" method of doing it, or am I missing something critical?

Quoted from "InvisiblePrawn"

What (if any) light does this shine on the acual in-game "Skill" stat. I know we've got the formula somewhere. From my experience, it tends to vary quite a bit. I imagine gamemode and similar factors have a fairly significant effect.

Skill correlates with SPM and KPM quite well (0.7 and 0.8 ), but with KD it's "only" 0.36. The formula is somewhere on the forums indeed, and I think there were at least one of these listed somewhere. I guess it has more weight on other-than-kdr variables.
• 3VerstsNorth - Analysis of game mechanics in BF4 (tickrates, effects of tickrate, etc)
• leptis - Analysis of shotguns, recoil, recoil control and air drag.
• Veritable - Scoring of BF4/BF1 firearms in terms of usability, firing and other mechanics.
• Miffyli - Random statistical analysis of BF4 battlereports/players and kill-distances. (list is cluttered with other threads).
Sorry if your name wasn't on the list, I honestly can't recall all names : ( . Nudge me if you want to be included

Posts: 42

Date of registration
: Apr 4th 2014

Platform: PC

Battlelog:

Reputation modifier: 2

Monday, April 7th 2014, 4:45pm

Quoted from "Miffyli"

I guess it has more weight on other-than-kdr variables.

I really like that. I guess the Skill may be more useful than I originally thought -- always seemed to be a "broken" metric to me.

Posts: 6

Date of registration
: Jan 22nd 2014

Platform: Xbox One

Battlelog:

Reputation modifier: 2

Monday, April 7th 2014, 5:20pm

Awesome work!

Could you possibly show the distribution of the other variables(time,kd, spm,etc)? The rank histogram was very useful, clearly shows what is "normal".
Stats

Spoiler

Symthic Developer

Posts: 3,694

Date of registration
: Mar 21st 2013

Platform: PC

Location: __main__, Finland

Reputation modifier: 16

Monday, April 7th 2014, 5:36pm

Quoted from "Gn3sda"

Awesome work!

Could you possibly show the distribution of the other variables(time,kd, spm,etc)? The rank histogram was very useful, clearly shows what is "normal".

Here you go. Gameplaytime, SPM and KPM. KD was acting weird for some reason.

more hists

Click for bigger.

Also five-number summaries as a copy-pasta:

Source code

1
2
3

SPM:     0.0000  328.0347  451.9802  619.1244  3232.8914
KPM:     0.0000  0.414393  0.5730724 0.7577548 3.0480124
Time(h): 0.0000  69.12722  128.91722 212.84333 1240.09444
• 3VerstsNorth - Analysis of game mechanics in BF4 (tickrates, effects of tickrate, etc)
• leptis - Analysis of shotguns, recoil, recoil control and air drag.
• Veritable - Scoring of BF4/BF1 firearms in terms of usability, firing and other mechanics.
• Miffyli - Random statistical analysis of BF4 battlereports/players and kill-distances. (list is cluttered with other threads).
Sorry if your name wasn't on the list, I honestly can't recall all names : ( . Nudge me if you want to be included

Data Analyzer

Posts: 7,863

Date of registration
: Dec 16th 2011

Platform: PC

Location: Sweden

Battlelog:

Reputation modifier: 20

Monday, April 7th 2014, 9:56pm

As any other "skill" stat, xp boosts and k/d padding / various game modes utterly destroy any chance of making anything out of stats related to them.. Heck skill can't even be measured unless people play at their best all the time.. Good players can just derp around if they feel like it so... uhh.... It's all really obvious so... Why bother going down the "skill" stat path? ;o

Quoted

(14:06:57) Riesig: I should stop now. People might get sig material again