Apache Spark has a useful command prompt interface but its true power comes from complex data pipelines that are run non-interactively. Implementing such pipelines can be a daunting task for anyone not familiar with the tools used to build and deploy application software. This article is meant show all the …
...read moreThere are comments.
Polar graphs can be a good way to represent cyclical data, such as traffic by day of week. In this post I take stack overflow data and plot the usage of tags throughout the week. The method also adds errors to the matplotlib polar plot as a shaded region to help understand the variability in the data. The data is available here ...read more
There are comments.
I recently switched DSLR camera systems from Canon to Nikon for reasons of marital harmony. That meant choosing which Nikon lenses would replace the four Canon lenses I owned. To make an optimal decision I needed to know my historical usage, so I wrote some python to analyze image metadata from 10 years of digital photography.
...read moreThere are comments.
The Timedelta object is relatively new to pandas. Recently I worked with Timedeltas but found it wasn't obvious how to do what I wanted. Worse, some operations were seemingly obvious but could easily return the wrong answer (update: this issue was fixed in pandas version 0.17.0). Here I go through a few Timedelta
...read more
There are comments.
Amazon Web Services recently announced Amazon Machine Learning, promising to make large scale machine learning more accessible to non-experts. I was curious to try out this service so I fed it some weather data from Oakland International Airport to see how well and …
...read moreThere are comments.
In part 3 …
...read moreThere are comments.
In this post …
...read moreThere are comments.
In an part …
...read moreThere are comments.
Source code for …
...read moreThere are comments.
In Part I of this post I made a topic model of the speech of Shakespeare characters from eight plays. Here in Part II I'll analyze the results of the model. Download notebook.
There are comments.
In this post I extract all the words spoken by each character in eight of Shakespeare's plays. Then I construct a topic model to see which characters are generally speaking about similar things. In Part II I look into the information revealed by the topic model. Download notebook.
There are comments.
I recently starting collecting data from the BART API, specifically estimated time to departure for trains at the two stations I use most frequently. In this notebook I'll show how I parsed the data from a csv file, reshaped it to fit the questions at hand, and made a few plots. Download notebook ...read more
There are comments.
Over the weekend I got curious about how different posts in this blog were similar to each other, and thought about putting links to similar posts at the end of each article. I used the gensim python library (topic modeling for humans) to find similar articles and I wrote a …
...read moreThere are comments.
Data science, I'm sorry to say, often involves cleaning up input data into a usable and uniform format. Command line tools like grep, awk and sed provide an arcane power to manipulate text in files of arbitrary size. Mastering these tools can separate data science novices from data scientists with …
There are comments.
A common need in data science is to test if a some group of data contains a given value. One specific example would be to test if a word is a stop word.
If the elements of the group exist in a list named group in python …
There are comments.
A friend of mine recently asked me to share some of my experiences in making the transition from a biophysics Ph.D. student to data scientist. I realized there are probably a lot of people interested in making a similar transition who could benefit from my experience.
A year …
...read moreThere are comments.
Page 1 / 1