Apache Spark has a useful command prompt interface but its true power comes from complex data pipelines that are run non-interactively. Implementing such pipelines can be a daunting task for anyone not familiar with the tools used to build and deploy application software. This article is meant show all the …
...read moreThere are comments.
There are times when it makes sense to offload work from a local laptop to a remote machine. When the data set already resides in AWS it's much faster to download it to an EC2 instance rather than to your machine. Occasionally a bit more RAM or disk space is …
...read moreThere are comments.
Polar graphs can be a good way to represent cyclical data, such as traffic by day of week. In this post I take stack overflow data and plot the usage of tags throughout the week. The method also adds errors to the matplotlib polar plot as a shaded region to help understand the variability in the data. The data is available here ...read more
There are comments.
Dates and times provide an unlimited source of hassles for anyone working with them. In this post I'll discuss a potential performance pitfall I encountered parsing dates in pandas. Conclusion: Create DatetimeIndices by parsing data with to_datetime(my_dates, format='my_format').
There are comments.
I recently switched DSLR camera systems from Canon to Nikon for reasons of marital harmony. That meant choosing which Nikon lenses would replace the four Canon lenses I owned. To make an optimal decision I needed to know my historical usage, so I wrote some python to analyze image metadata from 10 years of digital photography.
...read moreThere are comments.
The Timedelta object is relatively new to pandas. Recently I worked with Timedeltas but found it wasn't obvious how to do what I wanted. Worse, some operations were seemingly obvious but could easily return the wrong answer (update: this issue was fixed in pandas version 0.17.0). Here I go through a few Timedelta
...read more
There are comments.
I just noticed that GitHub now renders Jupyter (IPython) notebooks when you view the file. This is great! See GitHub's post
There are comments.
Amazon Web Services recently announced Amazon Machine Learning, promising to make large scale machine learning more accessible to non-experts. I was curious to try out this service so I fed it some weather data from Oakland International Airport to see how well and …
...read moreThere are comments.
Videos of the talks from PyCon 2015 can be found here. I haven't had a chance to watch any yet but I'm sure some of them will be excellent. You can see my favorite python and data science videos on the recommended videos page.
...read moreThere are comments.
Pandas 0.16.0 was released this week. Version 0.16.0 includes a new .assign() method of DataFrames, which I'll use on some real world automobile MPG data below (data file).
There are comments.
To show some techniques for working with files that are too large to fit on memory, I'm writing this post on a 10 year old laptop with 512 Mb of RAM and a 1.2 GHz celeron processor. The data in question is an xml format dump of data from …
...read moreThere are comments.
Spark 1.3.0 is out (release notes). I'm particularly excited about the new DataFrame API, it should allow much more natural code for many analytics tasks.
...read moreThere are comments.
IPython 3.0 has been released, with notable changes to the IPython Notebook, which is now evolving into the language-agnostic Project Jupyter.
I think some of the new nbconvert tools will be useful:
...read more
- Added a .ipynb exporter to nbconvert. It can be used by passing --to notebook as a commandline …
There are comments.
File compression tools like gzip and bzip2 can compress text files into a fraction of their size, often to as little as 20% of the original. Data files often come compressed to save storage space and network bandwidth. A typical workflow is to uncompress the file before analysis, but it can be more convenient to leave the file in its compressed form, especially if the uncompressed file would take up a significant amount of space. In this post I'll show how to work directly with compressed files in python.
There are comments.
In part 3 …
...read moreThere are comments.
I just posted a 4 part series on setting up a Flask app on EC2, filling out a database, and visualizing the data with D3.js. The original motivation for this project was to answer the question "What is the latest I can leave work while still having a 90 …
...read moreThere are comments.
In this post …
...read moreThere are comments.
In an part …
...read moreThere are comments.
Source code for …
...read moreThere are comments.
Page 1 / 2 »