Data Science

DataFramesMeta.jl adds dplyr-like macros:

36.1 DataFrames.jl

DataFramesMeta.jl adds dplyr-like macros:

Alternatives: Tidier.jl (tidyverse-style), Query.jl, TidierData.jl, DuckDB.jl (SQL), TypedTables.jl.

using DataFrames, CSV, Statistics

df = DataFrame(name=["Ann", "Bob", "Cara", "Dan"], age=[30, 25, 35, 28],
               city=["Oslo", "Rome", "Oslo", "Lima"], score=[88, 92, missing, 75])
size(df), names(df), describe(df), first(df, 2), last(df, 1)

df.age                                  # column vector
df[!, :age]                             # no copy; df[:, :age] copies
df[2, :]                                # a row (DataFrameRow)
df[df.age .> 26, [:name, :city]]        # filter + select
subset(df, :age => ByRow(>(26)))
select(df, :name, :age => (a -> a .* 12) => :age_months)
transform!(df, :score => ByRow(s -> coalesce(s, 0)) => :score_filled)
sort(df, [:city, order(:age, rev=true)])
rename!(df, :score => :points)
dropmissing(df, :points)

# Split-apply-combine
gdf = groupby(df, :city)
combine(gdf, :age => mean => :mean_age, nrow => :n)
combine(groupby(df, :city), :points => (x -> mean(skipmissing(x))) => :avg_points)

# Joins, reshaping, adding rows
other = DataFrame(city=["Oslo", "Rome"], country=["NO", "IT"])
leftjoin(df, other, on=:city); innerjoin(df, other, on=:city)
stack(df, [:age, :points]); unstack(long, :name, :variable, :value)
push!(df, ("Eve", 40, "Oslo", 99, 
…

using DataFramesMeta
@chain df begin
    @subset(:age .> 25)
    @transform(:age2 = :age .^ 2)
    @groupby(:city)
    @combine(:mean_age = mean(:age))
end

36.2 Reading & writing data

df = CSV.read("data.csv", DataFrame; missingstring="NA", dateformat="yyyy-mm-dd")
CSV.write("out.csv", df)
CSV.File("big.csv"; select=[:a, :b]) |> DataFrame
using XLSX; XLSX.readtable("file.xlsx", "Sheet1") |> DataFrame
using Arrow; Arrow.write("df.arrow", df); DataFrame(Arrow.Table("df.arrow"))
using Parquet2; DataFrame(Parquet2.Dataset("df.parquet"))
using JLD2; jldsave("data.jld2"; df); load("data.jld2", "df")
using ReadStatTables  # SPSS/Stata/SAS

36.3 Statistics & modeling

Also: MixedModels.jl (fast mixed-effects models), Survival.jl, Distributions.jl, StatsModels.jl, Bootstrap.jl, TimeSeries.jl, StateSpaceModels.jl, Loess.jl, KernelDensity.jl, CategoricalArrays.jl.

using Statistics, StatsBase, HypothesisTests, GLM

x = randn(100); y = 2x .+ randn(100)
mean(x), median(x), std(x), var(x), quantile(x, [0.25, 0.75]), cor(x, y), cov(x, y)
StatsBase.countmap(["a", "b", "a"]), mode([1, 2, 2]), zscore(x), skewness(x), kurtosis(x)
summarystats(x)

OneSampleTTest(x, 0.0) |> pvalue
EqualVarianceTTest(x, y); MannWhitneyUTest(x, y); ChisqTest([10 20; 30 40])

data = DataFrame(x=x, y=y)
ols = lm(@formula(y ~ x), data)
coef(ols), stderror(ols), r2(ols), predict(ols, DataFrame(x=[1.0]))
logit = glm(@formula(z ~ x), df, Binomial(), LogitLink())