https:\/\/gerrit.googlesource.com\/gerrit\/<\/a> – I was looking at the codebase from 02bafe0f4c51aa24b2b05d4d1309ecfc828762c0 (January 20th, 2016)<\/p>\nIndependence check<\/h2>\n
With the previous information – and the notion of a vector representation – I thought about the possibility to check for independence.<\/p>\n
If two vectors are independent, then they should be orthogonal. If two code bases are independent, then they should be orthogonal in their domain vectors. To test this, we can try to plot the words used in the code bases. Naturally, we would need to strip away the language keywords, but as we will see, this is not quite as necessary as expected. We can even gain other insights by looking at the keyword uses.<\/p>\n
So, as above, I created word frequence files for two JavaScript projects.<\/p>\n
p1 <- read.table(“p1-wordfreq.txt”, header=F)
\np2 <- read.table(“p2-wordfreq.txt”, header=F)<\/p>\n
We don’t really want the exact count, so we pick the relative frequencies<\/p>\n
p1$V1 <- p1$V1\/max(p1$V1)
\np2$V1 <- p2$V1\/max(p2$V1)<\/p>\n
Now, we only want to look at the tokens they have in common to see whether or not they are orthogonal – the tokens not common are already orthogonal.<\/p>\n
common <- merge(p1, p2, by = “V2”)<\/p>\n
plot(common$V1.x, common$V1.y, xlab=”p1″, ylab=”p2″, main=”Comparing p1 and p2″)<\/p>\n
![\"comparing](\"http:\/\/serverdude.dk\/wp-content\/uploads\/comparing-projects-p1-p2.png\" "\\"comparing")
<\/p>\n
Next, we want to identify the JavaScript keywords.<\/p>\n
js <- read.table(“JavaScriptKeywords.txt”, header=F)
\nnames(js) <- “V2″ # js is a single column, we want to merge on the keywords in the same column names
\njs2 <- merge(js, common, by=”V2″)
\npoints(js2$V1.x, js2$V1.y, pch=19, col=”red”)<\/p>\n
# mark the 20% in both directions, thus we get a Pareto segmentation
\nabline(h=.2, col=”blue”)
\nabline(v=.2, col=”blue”)<\/p>\n
high <- common[common$V1.x > .2 & common$V1.y > .2,]<\/p>\n
The most frequently used non-keywords:<\/p>\n
high[-(match(intersect(high$V2, js2$V2), high$V2)),]
\nV2 V1.x V1.y
\n34 data 0.4170306 0.2444444
\n49 err 0.5545852 0.4555556
\n50 error 0.3013100 0.8000000
\n115 censored 0.6812227 0.6888889
\n131 settings 0.2052402 0.2111111<\/p>\n
The second to last in this list has been censored, it does provide an indication that the projects aren’t quite independent. The error, err, and data are so common and nondescript that it is somewhat okay to find them in this area, though I’d rather have less callback functions and better names in general.<\/p>\n
The most frequently used keywords:<\/p>\n
high[(match(intersect(high$V2, js2$V2), high$V2)),]
\nV2 V1.x V1.y
\n47 else 0.3449782 0.4444444
\n65 function 1.0000000 0.8000000
\n72 if 0.4716157 0.5000000
\n154 var 1.0000000 0.6444444<\/p>\n
Again this can be explained by a lot of callbacks, which are often on the form:<\/p>\n
function(err, data) {
\nif(err){
\n} else {
\n}
\n}<\/p>\n
Another explanation could be lots of anonymous functions, though usually callback.<\/p>\n
Conclusion<\/h2>\n
Removing comments and imports should provide for a better picture of the code base. Even so, it seems to not exactly scream the domain or architecture.<\/p>\n
Bi-grams could be another improvement.<\/p>\n
Independence check of supposedly independent projects may reveal that they aren’t or that the code is skewed towards an unwanted design.<\/p>\n
It is far from perfect, but as always it brings a different way of looking at the code base, and it is relatively quick to do.<\/p>\n
Comparing large code bases somewhat defeats the purpose as regression to the mean tells nothing much of interest. Taking Gerrit as an example, then the most used token is “import”, which is used 27560 times and as we saw above, the interesting parts reveal themselves around 1100 uses, which is less than 4%.<\/p>\n
![\"comparing](\"http:\/\/serverdude.dk\/wp-content\/uploads\/comparing-gerrit-dotcms.png\" "\\"comparing")
![\"comparing](\"http:\/\/serverdude.dk\/wp-content\/uploads\/comparing-gerrit-dotcms-loglog.png\" "\\"comparing")
<\/p>\n
Comparing Gerrit and an old repo I had of dotCMS, we find that the most used keywords including entities in java.lang are:<\/p>\n
import
\nString
\npublic
\nreturn
\nif
\nnew
\nthis
\nnull
\nprivate
\nvoid
\nstatic<\/p>\n
Which could indicate a lot of String constants and conditional logic (with return statements instead of else clauses), and with a possibility of Primitive Obsession – well, the web does call for a lot of String use.<\/p>\n","protected":false},"excerpt":{"rendered":"
Does the code base scream the domain? I was wondering whether or not it would be possible to use parts of PageRank (https:\/\/en.wikipedia.org\/wiki\/PageRank) to gain insights into a code base. If PageRank works on web pages to ascertain what the contents of the page relates to, then likely a similar way could be construed for […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[6,10],"tags":[],"class_list":["post-330","post","type-post","status-publish","format-standard","hentry","category-programming","category-software-development"],"_links":{"self":[{"href":"https:\/\/serverdude.dk\/index.php?rest_route=\/wp\/v2\/posts\/330","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/serverdude.dk\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/serverdude.dk\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/serverdude.dk\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/serverdude.dk\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=330"}],"version-history":[{"count":4,"href":"https:\/\/serverdude.dk\/index.php?rest_route=\/wp\/v2\/posts\/330\/revisions"}],"predecessor-version":[{"id":339,"href":"https:\/\/serverdude.dk\/index.php?rest_route=\/wp\/v2\/posts\/330\/revisions\/339"}],"wp:attachment":[{"href":"https:\/\/serverdude.dk\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=330"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/serverdude.dk\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=330"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/serverdude.dk\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=330"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}
![\"gerrit](\"http:\/\/serverdude.dk\/wp-content\/uploads\/gerrit-loglog-plot.png\" "\\"gerrit")
<\/p>\n![\"gerrit](\"http:\/\/serverdude.dk\/wp-content\/uploads\/gerrit-loglog-lt-1100.png\" "\\"gerrit")
<\/p>\n