{"id":351,"date":"2022-08-28T19:48:38","date_gmt":"2022-08-28T17:48:38","guid":{"rendered":"http:\/\/serverdude.dk\/?p=351"},"modified":"2022-08-28T19:48:38","modified_gmt":"2022-08-28T17:48:38","slug":"benfords-law","status":"publish","type":"post","link":"https:\/\/serverdude.dk\/?p=351","title":{"rendered":"Benford’s Law"},"content":{"rendered":"\n

I was curious as to what I actually could do with Benford’s Law<\/a>. Could I detect anomalies that would warrant questioning? Where could I apply it?<\/p>\n\n\n\n

So – first off – what is Benford’s Law – or the Newcomb-Benford law? It is a natural occurring distribution if the domain spans several orders of magnitude. It states that 1 is then – by far – the most prevalent most significant digit. The theoretical frequency is found by applying log10(1+1\/digit).<\/p>\n\n\n\n

Let’s say our data set is based upon the number of people living in cities, then the cities with 10-19 million, 1 million, 100 -199 thousand, 10-19 thousand, 1 thousand, 100-199, 10-19, and 1 citizen would cover a little more than 30% of the cities, whereas cities with 20-29 million, 2 million, 200-299 thousand, 20-29 thousand, 2 thoushand, 200-299, 20-29, and 2 citizens would cover 17.6% of the cities.<\/p>\n\n\n\n

I find this counter intuitive, and thus very fascinating.<\/p>\n\n\n\n

I had a bank statement with deposits and withdrawal – 212 in total – I though, why not try to apply Benford’s law to that lot and see how well it fits, and if not, if there are any explanation.<\/p>\n\n\n\n

To get the most significant digit I chose to use the Log10 function basically writing the number in scientific notation. Then taking the exponent and divide the original amount to get a power of one, then finally take the absolute integer value:

abs(as.integer(amount\/10^floor(log10(abs(amount)))))<\/p>\n\n\n\n

Let’s assume the amount is 12345, then log10 would give us 4.091491, we then divide 12345 by 10^4 to get 1.2345, then take the integer value – not rounding – this gives us 1.<\/p>\n\n\n\n

Using dplyr in R to get the frequency for this, I used:<\/p>\n\n\n\n

data_hist <- as.data.frame(data %>% group_by(digit=abs(as.integer(amount\/10^floor(log10(abs(amount)))))) %>% summarize(count=n()) )<\/p>\n\n\n\n

Which gave me:<\/p>\n\n\n\n

\n
digit<\/th>count<\/th><\/tr>\n
1<\/td>54<\/td><\/tr>\n
2<\/td>29<\/td><\/tr>\n
3<\/td>42<\/td><\/tr>\n
4<\/td>18<\/td><\/tr>\n
5<\/td>24<\/td><\/tr>\n
6<\/td>15<\/td><\/tr>\n
7<\/td>11<\/td><\/tr>\n
8<\/td>6<\/td><\/tr>\n
9<\/td>13<\/td><\/tr>\n<\/tbody><\/table><\/figure>\n\n\n\n

To add the Benford values simply add the values:<\/p>\n\n\n\n

data_hist$benford = log10(1+1\/data_hist$digit)<\/p>\n\n\n\n

We would also like to have the frequency as a percentage, not as an observation count. <\/p>\n\n\n\n

data_hist$freq = data_hist$count \/ sum(data_hist$count)<\/p>\n\n\n\n

Now, we could use these two values to calculate which digits are the most off either simply using freq\/benford or (freq-benford)\/(freq+benford) – but let’s just plot the data as a bar chart along with the Benford curve.<\/p>\n\n\n\n

ggplot(data_hist, aes(x=digit, y =freq)) + geom_bar(stat=”identity”) + geom_line(aes(x=digit, y = benford)) + labs(title = “Bank payment most significant digit: Benfords law”) + scale_x_discrete(“digit”, data_hist$digit, waiver(), factor(data_hist$digit))<\/p>\n\n\n\n

\"\"<\/a><\/figure>\n\n\n\n

It would seem tha 3, 5, and 9 are overrepresented, which begs the question: Why?<\/p>\n\n\n\n

It turns out, that I have subscriptions in here and those are not random, and will thus skew my distribution. <\/p>\n\n\n\n

While this was relatively easy to do, and I did spot a few outliers, it seems I didn’t quite get what I was looking for. And the 4 subscriptions didn’t expose themselves in this plot – maybe if the other subscriptions had been removed. Studies for another day.<\/p>\n","protected":false},"excerpt":{"rendered":"

I was curious as to what I actually could do with Benford’s Law. Could I detect anomalies that would warrant questioning? Where could I apply it? So – first off – what is Benford’s Law – or the Newcomb-Benford law? It is a natural occurring distribution if the domain spans several orders of magnitude. It […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-351","post","type-post","status-publish","format-standard","hentry","category-ikke-kategoriseret"],"_links":{"self":[{"href":"https:\/\/serverdude.dk\/index.php?rest_route=\/wp\/v2\/posts\/351","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/serverdude.dk\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/serverdude.dk\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/serverdude.dk\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/serverdude.dk\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=351"}],"version-history":[{"count":1,"href":"https:\/\/serverdude.dk\/index.php?rest_route=\/wp\/v2\/posts\/351\/revisions"}],"predecessor-version":[{"id":355,"href":"https:\/\/serverdude.dk\/index.php?rest_route=\/wp\/v2\/posts\/351\/revisions\/355"}],"wp:attachment":[{"href":"https:\/\/serverdude.dk\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=351"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/serverdude.dk\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=351"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/serverdude.dk\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=351"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}