By Sharon Machlis, Executive Editor, Data & Analytics, Computerworld |

About |

The focus here is on data: from R tips to desktop tools to taking a hard look at data claims.

IBM, Cloudera join RStudio to create R interface to Apache Spark

R users can now use the popular dplyr package to tap into Apache Spark big data.

The new sparklyr package is a native dplyr interface to Spark, according to RStudio. After installing the package, users can "interactively manipulate Spark data using both dplyr and SQL (via DBI), according to an RStudio blog post, as well as "filter and aggregate Spark data sets then bring them into R for analysis and visualization." There is also access to Spark distributed machine-learning algorithms.

Connecting to a local Spark cluster would looking something like the code below, according to sparkly deployment documentation:


library(sparklyr)
sc <- spark_connect(master = "spark://local:7077")

And, running SQL with sparklyr might look like this:


library(DBI)
iris_preview <- dbGetQuery(sc, "SELECT * FROM iris LIMIT 10")

Sharon Machlis is Director of Editorial Data & Analytics at Foundry, where she works on data analysis and in-house editor tools in addition to writing. Her book Practical R for Mass Communication and Journalism was published by CRC Press. You can also find her on Mastodon at fosstodon.org/@smach

Bing’s AI chatbot came to work for me. I had to fire it.