Niimpy: behavioral data analysis¶

Niimpy is a Python package for analyzing and quantifying behavioral data. It uses pandas to read data from disk, perform basic manipulations, and provides many high-level functions for various types of data.

Introduction¶

What¶

Niimpy is a Python package for analyzing and quantifying behavioral data. It uses pandas to read data from disk, perform basic manipulations, provides explorative data analysis functions, offers many high-level preprocessing functions for various types of data, and has functions for behavioral data analysis.

For Who¶

Niimpy is intended for researchers and data scientists analyzing digital behavioral data. Its purpose is to facilitate data analysis by providing a standardized replicable workflow.

Why¶

Digital behavioral studies using personal digital devices typically produce rich multi-sensor longitudinal datasets of mixed data types. Analyzing such data requires multidisciplinary expertise and software designed for the purpose. Currently, no standardized workflow or tools exist to analyze such data sets. The analysis requires domain knowledge in multiple fields and programming expertise. Niimpy package is specifically designed to analyze longitudinal, multimodal behavioral data. Niimpy is a user-friendly open-source package that can be easily expanded and adapted to specific research requirements. The toolbox facilitates the analysis phase by providing tools for data management, preprocessing, feature extraction, and visualization. The more advanced analysis methods will be incorporated into the toolbox in the future.

How¶

The toolbox is divided into four layers by functionality: 1) reading, 2) preprocessing, 3) exploration, and 4) analysis. For more information about the layers, refer the toolbox Architecture and workflow chapter. The quick start guide is be a good place to start. More detailed demo Jupyter notebooks are provided in the user guide chapter. Instructions for individual functions can be found under API chapter niimpy package.

This documentation has following chapters:

Basic information about the toolbox
Quickstart guide
API documentation
User guide
Community guide
Data documentation

Basic information contain this introduction, installation instructions, software architecture and workflow schematics, and information about compatible data input-formats and the required data schema.

The quickstart guide provides a minimal working analysis example to get you started.

The API documentation has all technical details, containing instruction about how to use the toolbox functions, classes, return types, arguments and such.

The user guide provide more thorough examples of each toolbox layer functionalities. The examples are in Jupyter notebook format.

The community guide has information about the authors, community rules, contribution, and our collaborators.

Installation¶

Niimpy is a normal Python package to install. It is not currently available on PyPi, so you can install it manually from github repository:

pip install niimpy

Note: only supports Python 3 (tested on 3.6. and above).

Architecture and workflow¶

Niimpy toolbox functionality is organized into four layers:

Data Reading
Data Preprocessing
Data Exploration
Data Analysis.

Each layer in implemented as a module. Following table presents the layer properties.

Layer	Purpose
Reading	Read data from the on-disk formats
Preprocessing	Prepare data for analysis
Exploration	Initial analysis, explorative data analysis
Analysis	Data analysis

Layer: reading¶

Data is read from the on-disk formats.

Typical input consists of filenames on disk, and typical output is a pandas.DataFrame with a direct mapping of on-disk formats. For convenience, it may do various other small limiting and preprocessing, but should not look inside the data too much.

These are in niimpy.reading.

Layer: preprocessing¶

After reading the data for analysis, preprocessing can handle filtering, etc. using the standard schema columns. It does not look at or understand actual sensor values, and the unknown sensor-specific columns are passed straight through to a future layer.

Typical input arguments include the DataFrame, and output is the DataFrame slightly adjusted, without affecting sensor-specific columns.

These are in niimpy.preprocessing.

Layer: exploration¶

These functions can do data aggregation, basic analysis, and visualization which is not specific to any sensor, instead of to the data type.

These are in niimpy.exploration.

Layer: analysis¶

These functions understand the sensor values and perform analysis based on them.

These are often in modules specific to the type of analysis.

These are in niimpy.analysis.

Workflow¶

Typical behavioral data analysis workflow consists of following steps:

Data reading -> Preprocessing -> Explorations -> Analysis

Other possible workflows:

Data reading -> Exploration -> Preprocessing -> Analysis
Data reading -> Exploration -> Preprocessing -> Exploration -> Analysis

Niimpy workflow diagram

File formats¶

In principle, Niimpy can deal with any files of any format - you only need to convert them to a DataFrame. Still, it is very useful to have some common formats, so we present two standard formats with default readers:

CSV files are very standard and normal to create and understand, but in order to deal with them everything must be loaded into memory.
sqlite3 databases, which requires sqlite3 to read, but provides more power for filtering and automatic processing without reading everything into memory.

DataFrame format (in-memory)¶

In-memory, data is stored in a pandas DataFrame. This is basically a normal dataframe. There are some standardized columns (see the schema) and the index is a DatetimeIndex.

CSV files¶

CSV files should have a header that lists the column names and generally be readable by pandas.read_csv.

Reading these can be done with niimpy.read_csv:

[1]:

import os
import niimpy
import niimpy.config as config

# Read the battery data
df= niimpy.read_csv(config.MULTIUSER_AWARE_BATTERY_PATH, tz='Europe/Helsinki')

sqlite3 databases¶

For the purposes of niimpy, sqlite3 databases can generally be seen as supercharged CSV files.

A single database file could contain multiple datasets within it, thus when reading them a table name must be specified.

One reads the entire database into memory using sqlite.read_sqlite:

[2]:

# Read the sqlite3 data
df= niimpy.read_sqlite(config.SQLITE_SINGLEUSER_PATH, table="AwareScreen", tz='Europe/Helsinki')

You can list the tables within a database using niimpy.reading.read.read_sqlite_tables:

[3]:

niimpy.reading.read.read_sqlite_tables(config.SQLITE_SINGLEUSER_PATH)

[3]:

{'AwareScreen'}

sqlite3 files are highly recommended as a data storage format, since many common exploration options can be done within the database itself without reading the whole data into memory or writing an iterator. However, the interface is more difficult to use. Niimpy (before 2021-07) used this as its primary interface, but since then this interface has been de-emphasized. You can read more in the database section, but this is only recommended if you need efficiency when using massive amounts of data.

Other formats¶

You can add readers for any types of formats which you can convert into a Pandas dataframe (so basically anything). For examples of readers, see niimpy/reading/read.py. Apply the function niimpy.preprocessing.util.df_normalize in order to apply some standardizations to get the standard Niimpy format.

Data schema¶

This page documents the expected data schema of Niimpy. This does not extend to the contents of data from sensors (yet), but relates to the metadata applicable to all sensors.

By using a standardized schema (mainly column names), we can promote interoperability of various tools.

Format¶

Data is in a tabular (relational) format. A row is an observation, and columns are properties of observations. (At this level of abstraction, an “observation” may be one sensor observation, or some data which contains a package of multiple observations).

In Niimpy, this is internally stored and handled as a pandas.DataFrame. The schema naturally maps to the columns/rows of the DataFrames.

The on-disk format is currently irrelevant, as long as the producers can create a DataFrame of the necessary format. Currently, we provide readers for sqlite3 and csv. Other standards may be implemented later.

Standard columns in DataFrames¶

By having standard columns, we can create portable functions that easily operate on diverse data types.

The DataFrame index should be a pandas.DatetimeIndex.
user: opaque identifier for the user. Often a string or integer.
device: unique identifier for a user’s device (not the device type). For example, a user could have multiple phones, and each would have a separate device identifier.
time: timestamp of the observation, in unixtime (seconds since 00:00 on 1970-01-01), stored as an integer. Unixtime is a globally unique measure of an instance of time on Earth, and to get localtime it is combined with a timezone.

In on-disk formats, time is considered the master timestamp, many other time-based properties are computed from it (though you could produce your own DataFrames other ways). In some of the standard formats (CSV/sqlite3), when a file is read, this integer column is automatically converted to the datetime column below and the DataFrame index.
datetime: a DateTime-compatible object, such as in pandas a numpy.datetime64 object, used only in in-memory representations (not usually written to portable save files). This should be an timezone-aware object, and the data loader handles the timezone conversion. automatically added to DataFrames when loaded.

It is the responsibility of each loader (or preprocessor) to add this column to the in-memory representation by converting the time column to this format. This happens automatically with readers included in niimpy.
timezone: Timezone in some format. Not yet used, to be decided.
For questionaire data
- id: a question identifier. String, should be of form QUESTIONAIRE_QUESTION, for example PHQ9_01. The common prefix is used to group questions of the same series.
- answer: the answer to the question. Opaque identifier.

Sensor-specific schemas are defined elsewhere. Columns which are not defined here are allowed and considered to be part of the sensors, most APIs should pass through unknown columns for handling in a future layer (sensor analysis).

Other standard columns in Niimpy¶

These are not part of the primary schema, but are standard in Niimpy.

day: e.g. 2021-04-09 (str)
hour: hour of day, e.g. 15 (int)

Standard columns in on-disk formats¶

For the most part, this maps directly to the columns you see above. An on-disk format should have a time column (unixtime, integer) plus whatever else is needed for that particular sensor, based on the above.

How to cite¶

Ikäheimonen, A., Triana, A. M., Luong, N., Ziaei, A., Rantaharju, J., Darst, R., & Aledavood, T. (2023). Niimpy: a toolbox for behavioral data analysis. SoftwareX, 23, 101472.

bibtex format:

@article{niimpy,

title = {Niimpy: A toolbox for behavioral data analysis}, journal = {SoftwareX}, volume = {23}, pages = {101472}, year = {2023}, issn = {2352-7110}, doi = {https://doi.org/10.1016/j.softx.2023.101472}, url = {https://www.sciencedirect.com/science/article/pii/S2352711023001681}, author = {Arsi Ikäheimonen and Ana M. Triana and Nguyen Luong and Amirmohammad Ziaei and Jarno Rantaharju and Richard Darst and Talayeh Aledavood}, keywords = {Data analysis toolbox, Digital behavioral studies, Mobile sensing, Python package}, }

Quick start¶

We will guide you through the main features of niimpy. This guide assumes that you have basic knowledge of Python. Also, please refers to the installation page for installing niimpy.

This guide provides an example of reading and handling Aware battery data. The tutorial will guide you through 4 basic steps of a data analysis pipeline:

Reading
Preprocessing
Visualization
Basic analysis

[1]:

# Setting up plotly environment
import plotly.io as pio
pio.renderers.default = "png"

[2]:

import numpy as np
import niimpy
from niimpy import config
from niimpy.exploration.eda import punchcard, missingness
from niimpy.preprocessing import battery

Reading¶

niimpy provides a simple function to read data from csv and sqlite database. We will read a csv file containing 1 month of battery data from an individual.

[3]:

df = niimpy.read_csv(config.MULTIUSER_AWARE_BATTERY_PATH, tz='Europe/Helsinki')
df.head()

[3]:

	user	device	time	battery_level	battery_status	battery_health	battery_adaptor	datetime
2020-01-09 02:20:02.924999936+02:00	jd9INuQ5BBlW	3p83yASkOb_B	1.578529e+09	74	3	2	0	2020-01-09 02:20:02.924999936+02:00
2020-01-09 02:21:30.405999872+02:00	jd9INuQ5BBlW	3p83yASkOb_B	1.578529e+09	73	3	2	0	2020-01-09 02:21:30.405999872+02:00
2020-01-09 02:24:12.805999872+02:00	jd9INuQ5BBlW	3p83yASkOb_B	1.578529e+09	72	3	2	0	2020-01-09 02:24:12.805999872+02:00
2020-01-09 02:35:38.561000192+02:00	jd9INuQ5BBlW	3p83yASkOb_B	1.578530e+09	72	2	2	0	2020-01-09 02:35:38.561000192+02:00
2020-01-09 02:35:38.953000192+02:00	jd9INuQ5BBlW	3p83yASkOb_B	1.578530e+09	72	2	2	2	2020-01-09 02:35:38.953000192+02:00

Preprocessing¶

There are various ways to handle battery data. For example, you can extract the gaps between consecutive battery timestamps.

[4]:

gaps = battery.battery_gaps(df, {})
gaps.head()

[4]:

		battery_gap
user
iGyXetHE3S8u	2019-08-05 14:00:00+03:00	0 days 00:01:18.600000
	2019-08-05 14:30:00+03:00	0 days 00:27:18.396000
	2019-08-05 15:00:00+03:00	0 days 00:51:11.997000192
	2019-08-05 15:30:00+03:00	NaT
	2019-08-05 16:00:00+03:00	0 days 00:59:23.522999808

niimpy can also extract the amount of battery data found within an interval.

[5]:

occurences = battery.battery_occurrences(df, {"resample_args": {"rule": "1H"}})
occurences.head()

[5]:

		occurrences
user
iGyXetHE3S8u	2019-08-05 14:00:00+03:00	3
	2019-08-05 15:00:00+03:00	1
	2019-08-05 16:00:00+03:00	1
	2019-08-05 17:00:00+03:00	1
	2019-08-05 18:00:00+03:00	1

Visualization¶

niimpy provides a selection of visualization tools curated for exploring behavioural data. For example, you can examine the frenquency of battery level in specified interval.

[6]:

fig = missingness.bar_count(df, columns=['battery_level'], sampling_freq='T')
fig.show()

In addition, you can analyze the battery level at each sampling interval by using a punchcard plot.

[7]:

fig = punchcard.punchcard_plot(df,
                                   user_list=['jd9INuQ5BBlW'],
                                   columns=['battery_status', 'battery_level'],
                                   resample='10T',
                                   title="Battery level")
fig.show()

For more information, refer to the Exploration section.

niimpy API docs¶

This section provides function reference for Niimpy. Please refer to the user guide for further details on the function usage.

niimpy package¶

Subpackages¶

niimpy.analysis package¶

Module contents¶

niimpy.exploration package¶

Subpackages¶

niimpy.exploration.eda package¶

Submodules¶

niimpy.exploration.eda.categorical module¶

Created on Thu Nov 18 14:49:22 2021

@author: arsii

niimpy.exploration.eda.categorical.categorize_answers(df, question)[source]¶

Extract a question answered and count different answers.

Parameters

dfPandas Dataframe: Dataframe containing questionnaire data
questionstr: dataframe column sontaining question id
answer_columnstr: dataframe column containing the answer

Returns

category_counts: Pandas Dataframe: Dataframe containing the category counts of answers filtered by the question

niimpy.exploration.eda.categorical.get_xticks_(ser)[source]¶

Helper function for plot_categories function. Convert series index into xtick values and text.

Parameters

serPandas series: Series containing the categorized counts

niimpy.exploration.eda.categorical.plot_categories(df, title=None, xlabel=None, ylabel=None, width=900, height=900)[source]¶

Create a barplot of categorical data

Parameters

dfPandas Dataframe: Dataframe containing categorized data
titlestr: Plot title
xlabelstr: Plot xlabel
ylabelstr: Plot ylabel
widthinteger: Plot width
heightinteger: Plot height

Returns

fig: plotly Figure: A barplot of the input data

niimpy.exploration.eda.categorical.plot_grouped_categories(df, group, title=None, xlabel=None, ylabel=None, width=900, height=900)[source]¶

Plot summary barplot for questionnaire data.

Parameters

df: Pandas DataFrameGroupBy: A grouped dataframe containing categorical data
group: str: Column used to describe group
titlestr: Plot title
xlabelstr: Plot xlabel
ylabelstr: Plot ylabel
widthinteger: Plot width
heightinteger: Plot height

Returns

fig: plotly Figure: Figure containing barplots of the data in each group

niimpy.exploration.eda.categorical.question_by_group(df, question, group='group')[source]¶

Plot summary barplot for questionnaire data.

Parameters

dfPandas Dataframe: Dataframe containing questionnaire data
questionstr: question id
answer_columnstr: answer_column containing the answer
groupstr: group by this column

Returns

dfPandas DataFrameGroupBy: Dataframe a single answers column filtered by the question parameter and grouped by the group parameter

niimpy.exploration.eda.categorical.questionnaire_grouped_summary(df, question, group='group', title=None, xlabel=None, ylabel=None, width=900, height=900)[source]¶

Create a barplot of categorical data

Parameters

dfPandas Dataframe: Dataframe containing questionnaire data
questionstr: question id
titlestr: Plot title
xlabelstr: Plot xlabel
ylabelstr: Plot ylabel
userBool or str: If str, plot single user data If False, plot group level data
groupstr: group by this column

Returns

fig: plotly Figure: A barplot of the input data

niimpy.exploration.eda.categorical.questionnaire_summary(df, question, title=None, xlabel=None, ylabel=None, user=None, width=900, height=900)[source]¶

Plot summary barplot for questionnaire data.

Parameters

dfPandas Dataframe: Dataframe containing questionnaire data
questionstr: question id
titlestr: Plot title
xlabelstr: Plot xlabel
ylabelstr: Plot ylabel
userBool or str: If str, plot single user data If False, plot group level data

Returns

fig: plotly Figure: A barplot summary of the questionnaire

niimpy.exploration.eda.countplot module¶

Created on Mon Nov 8 14:42:18 2021

@author: arsii

niimpy.exploration.eda.countplot.barplot_(df, fig_title, xlabel, ylabel)[source]¶

Plot a barplot showing counts for each subjects

A dataframe must have columns named ‘user’, containing the user id’s, and ‘values’ containing the observation counts.

Parameters

dfPandas Dataframe: Dataframe containing the data
fig_titlestr: Plot title
xlabelstr: Plot xlabel
ylabelstr: Plot ylabel

Returns

niimpy.exploration.eda.countplot.boxplot_(df, fig_title, points='outliers', y='values', xlabel='Group', ylabel='Count', binning=False)[source]¶

Plot a boxplot

Parameters

dfPandas Dataframe: Dataframe containing the data
fig_titlestr: Plot title
pointsstr: If ‘all’, show all observations next to boxplots If ‘outliers’, show only outlying points The default is ‘outliers’
y: str: A dataframe column to plot
xlabelstr: Plot xlabel
ylabelstr: Plot ylabel

Returns

niimpy.exploration.eda.countplot.calculate_bins(df, binning)[source]¶

Calculate time index based bins for each observation in the dataframe.

Parameters

dfPandas DataFrame
binningstr
to_stringbool

Returns

binspandas period index

niimpy.exploration.eda.countplot.countplot(df, fig_title, plot_type='count', points='outliers', aggregation='group', user=None, column=None, binning=False)[source]¶

Create boxplot comparing groups or individual users.

Parameters

dfpandas DataFrame: A DataFrame to be visuliazed
fig_titlestr: The plot title.
plot_typestr: If ‘count’, plot observation count per group (boxplot) or by user (barplot) If ‘value’, plot observation values per group (boxplot) The default is ‘count’
aggregationstr: If ‘group’, plot group level summary If ‘user’, plot user level summary The default is ‘group’
userstr: if given … The default is None
columnstr, optional: if None, count number of rows. If given, count only occurances of that column. The default is None.

Returns

niimpy.exploration.eda.countplot.get_counts(df, aggregation)[source]¶

Calculate datapoint counts by group or by user

Parameters

dfPandas DataFrame
aggregationstr

Returns

n_eventsPandas DataFrame

niimpy.exploration.eda.lineplot module¶

Created on Wed Oct 27 09:53:46 2021

@author: arsii

niimpy.exploration.eda.lineplot.calculate_averages_(df, column, by)[source]¶: calculate group averages by given timerange

niimpy.exploration.eda.lineplot.plot_averages_(df, column, by='hour')[source]¶

Plot user group level averages by hour or by weekday.

Parameters

dfPandas Dataframe: Dataframe containing the data
columnstr: Columns to plot.
bystr, optional: Indicator for group level averaging. The default is False. If ‘hour’, hourly averages per group are presented. If ‘weekday’, daily averages per gruop are presented.

Returns

None.

niimpy.exploration.eda.lineplot.plot_timeseries_(df, columns, users, title, xlabel, ylabel, resample=False, interpolate=False, window_len=False, reset_index=False)[source]¶

There goes the text.

Parameters

dfPandas Dataframe: Dataframe containing the data
columnslist or str: Columns to plot.
userslist or str: Users to plot.
titlestr: Plot title.
xlabelstr: Plot xlabel.
ylabelstr: Plot ylabel.
resamplestr, optional: Data resampling frequency. The default is False. For details: https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.resample.html
interpolatebool, optional: If true, time series will be interpolated using splines. The default is False.
windowint, optional: Rolling window smoothing window size. The default is False.
reset_indexbool, optional: If true, dataframe index will be resetted. The default is False.

Returns

None.

niimpy.exploration.eda.lineplot.resample_data_(df, resample, interpolate, window_len, reset_index)[source]¶: resample dataframe for plotting

niimpy.exploration.eda.lineplot.timeplot(df, users, columns, title, xlabel, ylabel, resample=False, interpolate=False, window=False, reset_index=False, by=False)[source]¶

Plot a time series plot. Plot selected users and columns or group level averages, aggregated by hour or weekday.

Parameters

dfPandas Dataframe: Dataframe containing the data
userslist or str: Users to plot.
columnslist or str: Columns to plot.
titlestr: Plot title.
xlabelstr: Plot xlabel.
ylabelstr: Plot ylabel.
resamplestr, optional: Data resampling frequency. The default is False. For details: https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.resample.html
interpolatebool, optional: If true, time series will be interpolated using splines. The default is False.
windowint, optional: Rolling window smoothing window size. The default is False.
reset_indexbool, optional: If true, dataframe index will be resetted. The default is False.
bystr, optional: Indicator for group level averaging. The default is False. If ‘hour’, hourly averages per group are presented. If ‘weekday’, daily averages per gruop are presented.
Returns
——-
None.

niimpy.exploration.eda.missingness module¶

This module is rewritten based on the missingno package. The original files can be found here: https://github.com/ResidentMario/missingno

niimpy.exploration.eda.missingness.bar(df, columns=None, title='Data frequency', xaxis_title='', yaxis_title='', sampling_freq=None, sampling_method='mean')[source]¶

Display bar chart visualization of the nullity of the given DataFrame.

Parameters

df: pandas Dataframe: Dataframe to plot
columns: list, optional: Columns from input dataframe to investigate missingness. If none is given, uses all columns.
title: str: Figure’s title
xaxis_title: str, optional: x_axis’s label
yaxis_title: str, optional: y_axis’s label
sampling_freq: str, optional: Frequency to resample the data. Requires the dataframe to have datetime-like index. Possible values: ‘H’, ‘T’
sampling_method: str, optional: Resampling method. Possible values: ‘sum’, ‘mean’. Default value is ‘mean’.
Returns
——-
fig: Plotly figure.

niimpy.exploration.eda.missingness.bar_count(df, columns=None, title='Data frequency', xaxis_title='', yaxis_title='', sampling_freq='H')[source]¶

Display bar chart visualization of the nullity of the given DataFrame.

Parameters

df: pandas Dataframe: Dataframe to plot
columns: list, optional: Columns from input dataframe to investigate missingness. If none is given, uses all columns.
title: str: Figure’s title
xaxis_title: str, optional: x_axis’s label
yaxis_title: str, optional: y_axis’s label
sampling_freq: str, optional: Frequency to resample the data. Requires the dataframe to have datetime-like index. Possible values: ‘H’, ‘T’

Returns

fig: Plotly figure.

niimpy.exploration.eda.missingness.heatmap(df, height=800, width=800, title='', xaxis_title='', yaxis_title='')[source]¶

Return ‘plotly’ heatmap visualization of the nullity correlation of the Dataframe.

Parameters

df: pandas Dataframe: Dataframe to plot
width: int:: Figure’s width
height: int:: Figure’s height
Returns
——-
fig: Plotly figure.

niimpy.exploration.eda.missingness.matrix(df, height=500, title='Data frequency', xaxis_title='', yaxis_title='', sampling_freq=None, sampling_method='mean')[source]¶

Return matrix visualization of the nullity of data. For now, this function assumes that the data frame is datetime indexed.

Parameters

df: pandas Dataframe: Dataframe to plot
columns: list, optional: Columns from input dataframe to investigate missingness. If none is given, uses all columns.
title: str: Figure’s title
xaxis_title: str, optional: x_axis’s label
yaxis_title: str, optional: y_axis’s label
sampling_freq: str, optional: Frequency to resample the data. Requires the dataframe to have datetime-like index. Possible values: ‘H’, ‘T’
sampling_method: str, optional: Resampling method. Possible values: ‘sum’, ‘mean’. Default value is ‘mean’.
Returns
——-
fig: Plotly figure.

niimpy.exploration.eda.punchcard module¶

Created on Thu Nov 18 16:14:47 2021

@author: arsii

niimpy.exploration.eda.punchcard.combine_dataframe_(df, user_list, columns, res, date_index, agg_func=<function mean>)[source]¶

resample values from multiple users into new dataframe

Parameters

dfPandas Dataframe: Dataframe containing the data
user_listlist: List containing user names/id’s (str)
columnslist: List of column names (str) to be plotted
resstr: Resample parameter e.g., ‘D’ for resampling by day
date_indexpd.date_range: Date range used as an index
agg_funcnumpy function: Aggregation function used with resample. The default is np.mean

Returns

df_combpd.DataFrame: Resampled and combined dataframe

niimpy.exploration.eda.punchcard.get_timerange_(df, resample)[source]¶

get first and last timepoint from the dataframe, and return a resampled datetimeindex.

Parameters

dfPandas Dataframe: Dataframe containing the data
ressamplestr: Resample parameter e.g., ‘D’ for resampling by day

Returns

date_indexpd.DatatimeIndex: Resampled DatetimeIndex

niimpy.exploration.eda.punchcard.punchcard_(df, title, n_xticks, xtitle, ytitle)[source]¶

create a punchcard plot

Parameters

dfPandas Dataframe: Dataframe containing the data
titlestr: Plot title.
n_xticksint or None: Number of xaxis ticks. If None, scaled automatically.
xtitlestr: Plot xaxis title
ytitlestr: Plot yaxis title

Returns

figplotly.graph_objs._figure.Figure: Punchcard plot

niimpy.exploration.eda.punchcard.punchcard_plot(df, user_list=None, columns=None, title='Punchcard Plot', resample='D', normalize=False, agg_func=<function mean>, timerange=False)[source]¶

Punchcard plot for given users and column with optional resampling

Parameters

dfPandas Dataframe: Dataframe containing the data
user_listlist, optional: List containing user id’s as string. The default is None.
columnslist, optional: List containing columns as strings. The default is None.
titlestr, optional: Plot title. The default is “Punchcard Plot”.
resamplestr, optional: Indicator for resampling frequency. The default is ‘D’ (day).
agg_funcnumpy function: Aggregation function used with resample. The default is np.mean
normalizeboolean, optional: If true, data is normalized using min-max-scaling. The default is False.
timerangeboolean or tuple, optional: If false, timerange is not filtered. If tuple containing timestamps, timerange is filtered. The default is False.

Returns

figplotly.graph_objs._figure.Figure: Punchcard plot

Module contents¶

Submodules¶

niimpy.exploration.missingness module¶

niimpy.exploration.missingness.missing_data_format(question, keep_values=False)[source]¶

Returns a series of timestamps in the right format to allow missing data visualization .

Parameters

question: Dataframe

niimpy.exploration.missingness.missing_noise(database, subject, start=None, end=None)[source]¶

Returns a Dataframe with the estimated missing data from the ambient noise sensor.

NOTE: This function aggregates data by day.

Parameters

database: Niimpy database
user: string
start: datetime, optional
end: datetime, optional

Returns

avg_noise: Dataframe

niimpy.exploration.missingness.screen_missing_data(database, subject, start=None, end=None)[source]¶

Returns a DataFrame contanining the percentage (range [0,1]) of loss data calculated based on the transitions of screen status. In general, if screen_status(t) == screen_status(t+1), we declared we have at least one missing point.

Parameters

database: Niimpy database
user: string
start: datetime, optional
end: datetime, optional

Returns

count: Dataframe

niimpy.exploration.setup_dataframe module¶

niimpy.exploration.setup_dataframe.create_categorical_dataframe()[source]¶

Create a sample Pandas dataframe used by the test functions.

Returns

dfpandas.DataFrame: Pandas dataframe containing sample data.

niimpy.exploration.setup_dataframe.create_dataframe()[source]¶

Create a sample Pandas dataframe used by the test functions.

Returns

dfpandas.DataFrame: Pandas dataframe containing sample data.

niimpy.exploration.setup_dataframe.create_missing_dataframe(nrows, ncols, density=0.9, random_state=None, index_type=None, freq=None)[source]¶

Create a Pandas dataframe with random missingness.

Parameters

nrowsint: Number of rows
ncolsint: Number of columns
density: float: Amount of available data
random_state: float, optional: Random seed. If not given, default to 33.
index_type: float, optional: Accepts the following values: “dt” for timestamp, “int” for integer.
freq: string, optional:: Sampling frequency. This option is only available is index_type is “dt”.

Returns

dfpandas.DataFrame: Pandas dataframe containing sample data with random missing rows.

niimpy.exploration.setup_dataframe.create_timeindex_dataframe(nrows, ncols, random_state=None, freq=None)[source]¶

Create a datetime index Pandas dataframe

Parameters

nrowsint: Number of rows
ncolsint: Number of columns
random_state: float, optional: Random seed. If not given, default to 33.
freq: string, optional:: Sampling frequency.
Returns
——-
dfpandas.DataFrame: Pandas dataframe containing sample data with random missing rows.

Module contents¶

niimpy.preprocessing package¶

Submodules¶

niimpy.preprocessing.application module¶

niimpy.preprocessing.application.app_count(df, bat, screen, config={})[source]¶

This function returns the number of times each app group has been used, within the specified timeframe. The app groups are defined as a dictionary within the config variable. Examples of app groups are social media, sports, games, etc. If no mapping is given, a default one will be used. If no resampling window is given, the function sets a 30 min default time window. The function aggregates the duration by user, by app group, by timewindow.

Parameters

df: pandas.DataFrame: Input data frame
bat: pandas.DataFrame: Dataframe with the battery information. If no data is available, an empty dataframe should be passed.
screen: pandas.DataFrame: Dataframe with the screen information. If no data is available, an empty dataframe should be passed.
config: dict, optional: Dictionary keys containing optional arguments for the computation of scrren information. Keys can be column names, other dictionaries, etc. The functions needs the column name where the data is stored; if none is given, the default name “” will be used. To include information about the resampling window, please include the selected parameters from pandas.DataFrame.resample in a dictionary called resample_args.

Returns

result: dataframe: Resulting dataframe

niimpy.preprocessing.application.app_duration(df, bat, screen, config=None)[source]¶

This function returns the duration of use of different app groups, within the specified timeframe. The app groups are defined as a dictionary within the config variable. Examples of app groups are social media, sports, games, etc. If no mapping is given, a default one will be used. If no resampling window is given, the function sets a 30 min default time window. The function aggregates the duration by user, by app group, by timewindow.

Parameters

df: pandas.DataFrame: Input data frame
bat: pandas.DataFrame: Dataframe with the battery information. If no data is available, an empty dataframe should be passed.
screen: pandas.DataFrame: Dataframe with the screen information. If no data is available, an empty dataframe should be passed.
config: dict, optional: Dictionary keys containing optional arguments for the computation of scrren information. Keys can be column names, other dictionaries, etc. The functions needs the column name where the data is stored; if none is given, the default name “application_name” will be used. To include information about the resampling window, please include the selected parameters from pandas.DataFrame.resample in a dictionary called resample_args.

Returns

result: dataframe: Resulting dataframe

niimpy.preprocessing.application.classify_app(df, config)[source]¶

This function is a helper function for other screen preprocessing. The function classifies the screen events into the groups specified by group_map.

Parameters

df: pandas.DataFrame: Input data frame
config: dict, optional: Dictionary keys containing optional arguments for the computation of screen information. Keys can be column names, other dictionaries, etc. It can contain a dictionary called group_map, which has the mapping to define the app groups. Keys should be the app name, values are the app groups (e.g. ‘my_app’:’my_app_group’)

Returns

df: dataframe: Resulting dataframe

niimpy.preprocessing.application.extract_features_app(df, bat, screen, features=None)[source]¶

This function computes and organizes the selected features for application events. The function aggregates the features by user, by app group, by time window. If no time window is specified, it will automatically aggregate the features in 30 mins non-overlapping windows. If no group_map is provided, a default one will be used.

The complete list of features that can be calculated are: app_count, and app_duration.

Parameters

df: pandas.DataFrame: Input data frame
features: dict, optional: Dictionary keys contain the names of the features to compute. If none is given, all features will be computed.

Returns

result: dataframe: Resulting dataframe

niimpy.preprocessing.application.group_data(df)[source]¶: Group the dataframe by a standard set of columns listed in group_by_columns.

niimpy.preprocessing.application.reset_groups(df)[source]¶: Group the dataframe by a standard set of columns listed in group_by_columns.

niimpy.preprocessing.audio module¶

niimpy.preprocessing.audio.audio_count_loud(df_u, config=None)[source]¶

This function returns the number of times, within the specified timeframe, when there has been some sound louder than 70dB in the environment. If there is no specified timeframe, the function sets a 30 min default time window. The function aggregates this number by user, by timewindow.

Parameters

df_u: pandas.DataFrame: Input data frame
config: dict: Dictionary keys containing optional arguments for the computation of scrren information. Keys can be column names, other dictionaries, etc. The functions needs the column name where the data is stored; if none is given, the default name employed by Aware Framework will be used. To include information about the resampling window, please include the selected parameters from pandas.DataFrame.resample in a dictionary called resample_args.

Returns

result: dataframe: Resulting dataframe

niimpy.preprocessing.audio.audio_count_silent(df_u, config=None)[source]¶

This function returns the number of times, within the specified timeframe, when there has been some sound in the environment. If there is no specified timeframe, the function sets a 30 min default time window. The function aggregates this number by user, by timewindow.

Parameters

df_u: pandas.DataFrame: Input data frame
config: dict: Dictionary keys containing optional arguments for the computation of screen information. Keys can be column names, other dictionaries, etc. The functions needs the column name where the data is stored; if none is given, the default name employed by Aware Framework will be used. To include information about the resampling window, please include the selected parameters from pandas.DataFrame.resample in a dictionary called resample_args.

Returns

result: dataframe: Resulting dataframe

niimpy.preprocessing.audio.audio_count_speech(df_u, config=None)[source]¶

This function returns the number of times, within the specified timeframe, when there has been some sound between 65Hz and 255Hz in the environment that could be specified as speech. If there is no specified timeframe, the function sets a 30 min default time window. The function aggregates this number by user, by timewindow.

Parameters

df_u: pandas.DataFrame: Input data frame
config: dict: Dictionary keys containing optional arguments for the computation of scrren information. Keys can be column names, other dictionaries, etc. The functions needs the column name where the data is stored; if none is given, the default name employed by Aware Framework will be used. To include information about the resampling window, please include the selected parameters from pandas.DataFrame.resample in a dictionary called resample_args.

Returns

result: dataframe: Resulting dataframe

niimpy.preprocessing.audio.audio_max_db(df_u, config=None)[source]¶

This function returns the maximum decibels of the recorded audio snippets, within the specified timeframe. If there is no specified timeframe, the function sets a 30 min default time window. The function aggregates this number by user, by timewindow.

Parameters

df_u: pandas.DataFrame: Input data frame
config: dict: Dictionary keys containing optional arguments for the computation of scrren information. Keys can be column names, other dictionaries, etc. The functions needs the column name where the data is stored; if none is given, the default name employed by Aware Framework will be used. To include information about the resampling window, please include the selected parameters from pandas.DataFrame.resample in a dictionary called resample_args.

Returns

result: dataframe: Resulting dataframe

niimpy.preprocessing.audio.audio_max_freq(df_u, config=None)[source]¶

This function returns the maximum frequency of the recorded audio snippets, within the specified timeframe. If there is no specified timeframe, the function sets a 30 min default time window. The function aggregates this number by user, by timewindow.

Parameters

df_u: pandas.DataFrame: Input data frame
config: dict: Dictionary keys containing optional arguments for the computation of scrren information. Keys can be column names, other dictionaries, etc. The functions needs the column name where the data is stored; if none is given, the default name employed by Aware Framework will be used. To include information about the resampling window, please include the selected parameters from pandas.DataFrame.resample in a dictionary called resample_args.

Returns

result: dataframe: Resulting dataframe

niimpy.preprocessing.audio.audio_mean_db(df_u, config=None)[source]¶

This function returns the mean decibels of the recorded audio snippets, within the specified timeframe. If there is no specified timeframe, the function sets a 30 min default time window. The function aggregates this number by user, by timewindow.

Parameters

df_u: pandas.DataFrame: Input data frame
config: dict: Dictionary keys containing optional arguments for the computation of scrren information. Keys can be column names, other dictionaries, etc. The functions needs the column name where the data is stored; if none is given, the default name employed by Aware Framework will be used. To include information about the resampling window, please include the selected parameters from pandas.DataFrame.resample in a dictionary called resample_args.

Returns

result: dataframe: Resulting dataframe

niimpy.preprocessing.audio.audio_mean_freq(df_u, config=None)[source]¶

This function returns the mean frequency of the recorded audio snippets, within the specified timeframe. If there is no specified timeframe, the function sets a 30 min default time window. The function aggregates this number by user, by timewindow.

Parameters

df_u: pandas.DataFrame: Input data frame
config: dict: Dictionary keys containing optional arguments for the computation of scrren information. Keys can be column names, other dictionaries, etc. The functions needs the column name where the data is stored; if none is given, the default name employed by Aware Framework will be used. To include information about the resampling window, please include the selected parameters from pandas.DataFrame.resample in a dictionary called resample_args.

Returns

result: dataframe: Resulting dataframe

niimpy.preprocessing.audio.audio_median_db(df_u, config)[source]¶

This function returns the median decibels of the recorded audio snippets, within the specified timeframe. If there is no specified timeframe, the function sets a 30 min default time window. The function aggregates this number by user, by timewindow.

Parameters

df_u: pandas.DataFrame: Input data frame
config: dict: Dictionary keys containing optional arguments for the computation of scrren information. Keys can be column names, other dictionaries, etc. The functions needs the column name where the data is stored; if none is given, the default name employed by Aware Framework will be used. To include information about the resampling window, please include the selected parameters from pandas.DataFrame.resample in a dictionary called resample_args.

Returns

result: dataframe: Resulting dataframe

niimpy.preprocessing.audio.audio_median_freq(df_u, config=None)[source]¶

This function returns the median frequency of the recorded audio snippets, within the specified timeframe. If there is no specified timeframe, the function sets a 30 min default time window. The function aggregates this number by user, by timewindow.

Parameters

df_u: pandas.DataFrame: Input data frame
config: dict: Dictionary keys containing optional arguments for the computation of scrren information. Keys can be column names, other dictionaries, etc. The functions needs the column name where the data is stored; if none is given, the default name employed by Aware Framework will be used. To include information about the resampling window, please include the selected parameters from pandas.DataFrame.resample in a dictionary called resample_args.

Returns

result: dataframe: Resulting dataframe

niimpy.preprocessing.audio.audio_min_db(df_u, config=None)[source]¶

This function returns the minimum decibels of the recorded audio snippets, within the specified timeframe. If there is no specified timeframe, the function sets a 30 min default time window. The function aggregates this number by user, by timewindow.

Parameters

df_u: pandas.DataFrame: Input data frame
config: dict: Dictionary keys containing optional arguments for the computation of scrren information. Keys can be column names, other dictionaries, etc. The functions needs the column name where the data is stored; if none is given, the default name employed by Aware Framework will be used. To include information about the resampling window, please include the selected parameters from pandas.DataFrame.resample in a dictionary called resample_args.

Returns

result: dataframe: Resulting dataframe

niimpy.preprocessing.audio.audio_min_freq(df_u, config=None)[source]¶

This function returns the minimum frequency of the recorded audio snippets, within the specified timeframe. If there is no specified timeframe, the function sets a 30 min default time window. The function aggregates this number by user, by timewindow.

Parameters

df_u: pandas.DataFrame: Input data frame
config: dict: Dictionary keys containing optional arguments for the computation of scrren information. Keys can be column names, other dictionaries, etc. The functions needs the column name where the data is stored; if none is given, the default name employed by Aware Framework will be used. To include information about the resampling window, please include the selected parameters from pandas.DataFrame.resample in a dictionary called resample_args.

Returns

result: dataframe: Resulting dataframe

niimpy.preprocessing.audio.audio_std_db(df_u, config=None)[source]¶

This function returns the standard deviation of the decibels of the recorded audio snippets, within the specified timeframe. If there is no specified timeframe, the function sets a 30 min default time window. The function aggregates this number by user, by timewindow.

Parameters

df_u: pandas.DataFrame: Input data frame
config: dict: Dictionary keys containing optional arguments for the computation of scrren information. Keys can be column names, other dictionaries, etc. The functions needs the column name where the data is stored; if none is given, the default name employed by Aware Framework will be used. To include information about the resampling window, please include the selected parameters from pandas.DataFrame.resample in a dictionary called resample_args.

Returns

result: dataframe: Resulting dataframe

niimpy.preprocessing.audio.audio_std_freq(df_u, config=None)[source]¶

This function returns the standard deviation of the frequency of the recorded audio snippets, within the specified timeframe. If there is no specified timeframe, the function sets a 30 min default time window. The function aggregates this number by user, by timewindow.

Parameters

df_u: pandas.DataFrame: Input data frame
config: dict: Dictionary keys containing optional arguments for the computation of scrren information. Keys can be column names, other dictionaries, etc. The functions needs the column name where the data is stored; if none is given, the default name employed by Aware Framework will be used. To include information about the resampling window, please include the selected parameters from pandas.DataFrame.resample in a dictionary called resample_args.

Returns

result: dataframe: Resulting dataframe

niimpy.preprocessing.audio.extract_features_audio(df, features=None)[source]¶

This function computes and organizes the selected features for audio snippets that have been recorded using Aware Framework. The function aggregates the features by user, by time window. If no time window is specified, it will automatically aggregate the features in 30 mins non-overlapping windows.

The complete list of features that can be calculated are: audio_count_silent, audio_count_speech, audio_count_loud, audio_min_freq, audio_max_freq, audio_mean_freq, audio_median_freq, audio_std_freq, audio_min_db, audio_max_db, audio_mean_db, audio_median_db, audio_std_db

Parameters

df: pandas.DataFrame: Input data frame
features: dict, optional: Dictionary keys contain the names of the features to compute. If none is given, all features will be computed.

Returns

result: dataframe: Resulting dataframe

niimpy.preprocessing.audio.group_data(df)[source]¶: Group the dataframe by a standard set of columns listed in group_by_columns.

niimpy.preprocessing.audio.reset_groups(df)[source]¶: Group the dataframe by a standard set of columns listed in group_by_columns.

niimpy.preprocessing.battery module¶

niimpy.preprocessing.battery.battery_charge_discharge(df, config)[source]¶: Returns a DataFrame showing the mean difference in battery values and mean battery charge/discharge rate within specified time windows. If there is no specified timeframe, the function sets a 30 min default time window. Parameters ———- df: dataframe with date index

niimpy.preprocessing.battery.battery_discharge(df, config)[source]¶

This function returns the mean discharge rate of the battery within a specified time window. If there is no specified timeframe, the function sets a 30 min default time window. The function aggregates this number by user, by timewindow. Parameters ———- df: pandas.DataFrame

Dataframe with the battery information

config: dict, optional: Dictionary keys containing optional arguments for the computation of scrren information. Keys can be column names, other dictionaries, etc.

Returns¶

result: dataframe

niimpy.preprocessing.battery.battery_gaps(df, config)[source]¶

Returns a DataFrame with the mean time difference between consecutive battery timestamps. The mean is calculated within intervals specified in config. The minimum size of the considered deltas can be decided with the min_duration_between parameter.

Parameters

df: pandas.DataFrame: Dataframe with the battery information
config: dict, optional: Dictionary keys containing optional arguments for the computation of batter information. Keys can be column names, other dictionaries, etc.
Optional arguments in config:: min_duration_between: Timedelta, for example, pd.Timedelta(minutes=5)

niimpy.preprocessing.battery.battery_mean_level(df, config)[source]¶

This function returns the mean battery level within the specified timeframe. If there is no specified timeframe, the function sets a 30 min default time window. The function aggregates this number by user, by timewindow. Parameters ———- df: pandas.DataFrame

Dataframe with the battery information

config: dict, optional: Dictionary keys containing optional arguments for the computation of scrren information. Keys can be column names, other dictionaries, etc.

Returns¶

result: dataframe

niimpy.preprocessing.battery.battery_median_level(df, config)[source]¶

This function returns the median battery level within the specified timeframe. If there is no specified timeframe, the function sets a 30 min default time window. The function aggregates this number by user, by timewindow. Parameters ———- df: pandas.DataFrame

Dataframe with the battery information

config: dict, optional: Dictionary keys containing optional arguments for the computation of scrren information. Keys can be column names, other dictionaries, etc.

Returns¶

result: dataframe

niimpy.preprocessing.battery.battery_occurrences(df, config)[source]¶

Returns a dataframe showing the amount of battery data points found within a specified time window. If there is no specified timeframe, the function sets a 30 min default time window. Parameters ———- df: pandas.DataFrame

Dataframe with the battery information

config: dict, optional: Dictionary keys containing optional arguments for the computation of batter information. Keys can be column names, other dictionaries, etc.

niimpy.preprocessing.battery.battery_shutdown_time(df, config)[source]¶

This function returns the total time the phone has been turned off within a specified time window. If there is no specified timeframe, the function sets a 30 min default time window. The function aggregates this number by user, by timewindow. Parameters ———- df: pandas.DataFrame

Dataframe with the battery information

config: dict, optional: Dictionary keys containing optional arguments for the computation of scrren information. Keys can be column names, other dictionaries, etc.

Returns¶

result: dataframe

niimpy.preprocessing.battery.battery_std_level(df, config)[source]¶

This function returns the standard deviation battery level within the specified timeframe. If there is no specified timeframe, the function sets a 30 min default time window. The function aggregates this number by user, by timewindow. Parameters ———- df: pandas.DataFrame

Dataframe with the battery information

config: dict, optional: Dictionary keys containing optional arguments for the computation of scrren information. Keys can be column names, other dictionaries, etc.

Returns¶

result: dataframe

niimpy.preprocessing.battery.extract_features_battery(df, features=None)[source]¶

Calculates battery features

Parameters

dfpd.DataFrame: dataframe of battery data. It must contain these columns: battery_level and battery_status.
featuresmap (dictionary) of functions that compute features.: it is a map of map, where the keys to the first map is the name of functions that compute features and the nested map contains the keyword arguments to that function. If there is no arguments use an empty map. Default is None. If None, all the available functions are used. Those functions are in the dict battery.ALL_FEATURES. You can implement your own function and use it instead or add it to the mentioned map.

Returns

featurespd.DataFrame: Dataframe of computed features where the index is users and columns are the the features.

niimpy.preprocessing.battery.find_battery_gaps(battery_df, other_df, config)[source]¶: Returns a dataframe showing the gaps found only in the battery data. The default interval is 6 hours. Parameters ———- battery_df: Dataframe other_df: Dataframe

The data you want to compare with

niimpy.preprocessing.battery.find_non_battery_gaps(battery_df, other_df, config)[source]¶: Returns a dataframe showing the gaps found only in the other data. The default interval is 6 hours. Parameters ———- battery_df: Dataframe other_df: Dataframe

The data you want to compare with

niimpy.preprocessing.battery.find_real_gaps(battery_df, other_df, config)[source]¶: Returns a dataframe showing the gaps found both in the battery data and the other data. The default interval is 6 hours. Parameters ———- battery_df: Dataframe other_df: Dataframe

The data you want to compare with

niimpy.preprocessing.battery.format_battery_data(df, config)[source]¶: Returns a DataFrame with battery data for a user. Parameters ———- battery: DataFrame with battery data

niimpy.preprocessing.battery.group_data(df)[source]¶: Group the dataframe by a standard set of columns listed in group_by_columns.

niimpy.preprocessing.battery.reset_groups(df)[source]¶: Group the dataframe by a standard set of columns listed in group_by_columns.

niimpy.preprocessing.battery.shutdown_info(df, config)[source]¶

Returns a pandas DataFrame with battery information for the timestamps when the phone has shutdown. This includes both events, when the phone has shut down and when the phone has been rebooted. NOTE: This is a helper function created originally to preprocess the application info data Parameters ———- bat: pandas.DataFrame

Dataframe with the battery information

config: dict, optional: Dictionary keys containing optional arguments for the computation of scrren information. Keys can be column names, other dictionaries, etc.

Returns¶

shutdown: pandas series

niimpy.preprocessing.communication module¶

niimpy.preprocessing.communication.call_count(df, config=None)[source]¶

This function returns the number of times, within the specified timeframe, when a call has been received, missed, or initiated. If there is no specified timeframe, the function sets a 30 min default time window. The function aggregates this number by user, by timewindow.

Parameters

df: pandas.DataFrame: Input data frame
config: dict: Dictionary keys containing optional arguments for the computation of scrren information. Keys can be column names, other dictionaries, etc. The functions needs the column name where the data is stored; if none is given, the default name employed by Aware Framework will be used. To include information about the resampling window, please include the selected parameters from pandas.DataFrame.resample in a dictionary called resample_args.

Returns

result: dataframe: Resulting dataframe

niimpy.preprocessing.communication.call_duration_mean(df, config=None)[source]¶

This function returns the average duration of each call type, within the specified timeframe. The call types are incoming, outgoing, and missed. If there is no specified timeframe, the function sets a 30 min default time window. The function aggregates this number by user, by timewindow.

Parameters

df: pandas.DataFrame: Input data frame
config: dict: Dictionary keys containing optional arguments for the computation of scrren information. Keys can be column names, other dictionaries, etc. The functions needs the column name where the data is stored; if none is given, the default name employed by Aware Framework will be used. To include information about the resampling window, please include the selected parameters from pandas.DataFrame.resample in a dictionary called resample_args.

Returns

result: dataframe: Resulting dataframe

niimpy.preprocessing.communication.call_duration_median(df, config=None)[source]¶

This function returns the median duration of each call type, within the specified timeframe. The call types are incoming, outgoing, and missed. If there is no specified timeframe, the function sets a 30 min default time window. The function aggregates this number by user, by timewindow.

Parameters

df: pandas.DataFrame: Input data frame
bat: pandas.DataFrame: Dataframe with the battery information
config: dict: Dictionary keys containing optional arguments for the computation of scrren information. Keys can be column names, other dictionaries, etc. The functions needs the column name where the data is stored; if none is given, the default name employed by Aware Framework will be used. To include information about the resampling window, please include the selected parameters from pandas.DataFrame.resample in a dictionary called resample_args.

Returns

result: dataframe: Resulting dataframe

niimpy.preprocessing.communication.call_duration_std(df, config=None)[source]¶

This function returns the standard deviation of the duration of each call type, within the specified timeframe. The call types are incoming, outgoing, and missed. If there is no specified timeframe, the function sets a 30 min default time window. The function aggregates this number by user, by timewindow.

Parameters

df: pandas.DataFrame: Input data frame
config: dict: Dictionary keys containing optional arguments for the computation of scrren information. Keys can be column names, other dictionaries, etc. The functions needs the column name where the data is stored; if none is given, the default name employed by Aware Framework will be used. To include information about the resampling window, please include the selected parameters from pandas.DataFrame.resample in a dictionary called resample_args.

Returns

result: dataframe: Resulting dataframe

niimpy.preprocessing.communication.call_duration_total(df, config=None)[source]¶

This function returns the total duration of each call type, within the specified timeframe. The call types are incoming, outgoing, and missed. If there is no specified timeframe, the function sets a 30 min default time window. The function aggregates this number by user, by timewindow.

Parameters

df: pandas.DataFrame: Input data frame
config: dict: Dictionary keys containing optional arguments for the computation of scrren information. Keys can be column names, other dictionaries, etc. The functions needs the column name where the data is stored; if none is given, the default name employed by Aware Framework will be used. To include information about the resampling window, please include the selected parameters from pandas.DataFrame.resample in a dictionary called resample_args.

Returns

result: dataframe: Resulting dataframe

niimpy.preprocessing.communication.call_outgoing_incoming_ratio(df, config=None)[source]¶

This function returns the ratio of outgoing calls over incoming calls, within the specified timeframe. If there is no specified timeframe, the function sets a 30 min default time window. The function aggregates this number by user, by timewindow.

Parameters

df: pandas.DataFrame: Input data frame
config: dict: Dictionary keys containing optional arguments for the computation of scrren information. Keys can be column names, other dictionaries, etc. The functions needs the column name where the data is stored; if none is given, the default name employed by Aware Framework will be used. To include information about the resampling window, please include the selected parameters from pandas.DataFrame.resample in a dictionary called resample_args.

Returns

result: dataframe: Resulting dataframe

niimpy.preprocessing.communication.extract_features_comms(df, features=None)[source]¶

This function computes and organizes the selected features for calls and SMS events. The function aggregates the features by user, by time window. If no time window is specified, it will automatically aggregate the features in 30 mins non-overlapping windows.

The complete list of features that can be calculated are: call_duration_total, call_duration_mean, call_duration_median, call_duration_std, call_count, call_outgoing_incoming_ratio, sms_count

Parameters

df: pandas.DataFrame: Input data frame
features: dict, optional: Dictionary keys contain the names of the features to compute. If none is given, all features will be computed.

Returns

result: dataframe: Resulting dataframe

niimpy.preprocessing.communication.group_data(df)[source]¶: Group the dataframe by a standard set of columns listed in group_by_columns.

niimpy.preprocessing.communication.reset_groups(df)[source]¶: Group the dataframe by a standard set of columns listed in group_by_columns.

niimpy.preprocessing.communication.sms_count(df, config=None)[source]¶

This function returns the number of times, within the specified timeframe, when an SMS has been sent/received. If there is no specified timeframe, the function sets a 30 min default time window. The function aggregates this number by user, by timewindow.

Parameters

df: pandas.DataFrame: Input data frame
config: dict: Dictionary keys containing optional arguments for the computation of scrren information. Keys can be column names, other dictionaries, etc. The functions needs the column name where the data is stored; if none is given, the default name employed by Aware Framework will be used. To include information about the resampling window, please include the selected parameters from pandas.DataFrame.resample in a dictionary called resample_args.

Returns

result: dataframe: Resulting dataframe

niimpy.preprocessing.filter module¶

Generic DataFrame filtering

This module provides functions for generic DataFrame filtering. In many cases, it is simpler to do these filtering operations yourself directly on the DataFrames, but these functions simplify the operations of standard arguments in other functions.

niimpy.preprocessing.filter.filter_dataframe(df, user=None, start=None, end=None, rename_columns={})[source]¶

Standard dataframe preprocessing filter.

This implements some standard and common dataframe preprocessing options, which are used in very many functions. It is likely simpler and more clear to do these yourself on the DataFrames directly.

select only certain user: df[‘user’] == user

select date range: df[start:end]

column map: df.rename(columns=rename_columns)

It returns a new dataframe (and does not modify the passed one in-place).

niimpy.preprocessing.location module¶

niimpy.preprocessing.location.cluster_locations(lats, lons, min_samples=5, eps=200)[source]¶

Performs clustering on the locations

Parameters

latspd.DataFrame: Latitudes
lonspd.DataFrame: Longitudes
mins_samplesint: Minimum number of samples to form a cluster. Default is 5.
epsfloat: Epsilone parameter in DBSCAN. The maximum distance between two neighbour samples. Default is 200.

Returns

clustersarray: Array of clusters. -1 indicates outlier.

niimpy.preprocessing.location.compute_nbin_maxdist_home(lats, lons, latlon_home, home_radius=50)[source]¶

Computes number of bins in home and maximum distance to home

Parameters

latspd.DataFrame: Latitudes
lonspd.DataFrame: Longitudes
latlon_homearray: A tuple (lat, lon) showing the coordinate of home

Returns

(n_home, max_dist_home)tuple: n_home: number of bins the person has been near the home max_dist_home: maximum distance that the person has been from home

niimpy.preprocessing.location.distance_matrix(lats, lons)[source]¶

Compute distance matrix using great-circle distance formula

https://en.wikipedia.org/wiki/Great-circle_distance#Formulae

Parameters

latsarray: Latitudes
lonsarray: Longitudes

Returns

distsmatrix: Entry (i, j) shows the great-circle distance between point i and j, i.e. distance between (lats[i], lons[i]) and (lats[j], lons[j]).

niimpy.preprocessing.location.extract_features_location(df, features=None)[source]¶

Calculates location features

Parameters

dfpd.DataFrame: dataframe of location data. It must contain these columns: double_latitude, double_longitude, user, group. double_speed is optional. If not provided, it will be computed manually.
speed_thresholdfloat: Bins whose speed is lower than speed_threshold are considred static and the rest are moving.
featuresmap (dictionary) of functions that compute features.: it is a map of map, where the keys to the first map is the name of functions that compute features and the nested map contains the keyword arguments to that function. If there is no arguments use an empty map. Default is None. If None, all the available functions are used. Those functions are in the dict location.ALL_FEATURES. You can implement your own function and use it instead or add it to the mentioned map.

Returns

featurespd.DataFrame: Dataframe of computed features where the index is users and columns are the the features.

niimpy.preprocessing.location.filter_location(location, remove_disabled=True, remove_zeros=True, remove_network=True)[source]¶

Remove low-quality or weird location samples

Parameters

locationpd.DataFrame: DataFrame of locations
remove_disabledbool: Remove locations whose label is disabled
remove_zersobool: Remove locations which their latitude and longitueds are close to 0
remove_networkbool: Keep only locations whose provider is gps

Returns

locationpd.DataFrame

niimpy.preprocessing.location.find_home(lats, lons, times)[source]¶

Find coordinates of the home of a person

Home is defined as the place most visited between 12am - 6am. Locations within this time period first clustered and then the center of largest clusetr shows the home.

Parameters

latsarray-like: Latitudes
lonsarray-like: Longitudes
timesarray-like: Time of the recorderd coordinates
Returns
——
(lat_home, lon_home)tuple of floats: Coordinates of the home

niimpy.preprocessing.location.get_speeds_totaldist(lats, lons, times)[source]¶

Computes speed of bins with dividing distance by their time difference

Parameters

latsarray-like: Array of latitudes
lonsarray-like: Array of longitudes
timesarray-like: Array of times associted with bins
Returns
——
(speeds, total_distances)tuple of speeds (array) and total distance travled (float)

niimpy.preprocessing.location.group_data(df)[source]¶: Group the dataframe by a standard set of columns listed in group_by_columns.

niimpy.preprocessing.location.location_distance_features(df, config={})[source]¶

Calculates features related to distance and speed.

Parameters

df: dataframe with date index
config: A dictionary of optional arguments
Optional arguments in config:: longitude_column: The name of the column with longitude data in a floating point format. Defaults to ‘double_longitude’. latitude_column: The name of the column with latitude data in a floating point format. Defaults to ‘double_latitude’. speed_column: The name of the column with speed data in a floating point format. Defaults to ‘double_speed’. resample_args: a dictionary of arguments for the Pandas resample function. For example to resample by hour, you would pass {“rule”: “1H”}.

niimpy.preprocessing.location.location_number_of_significant_places(df, config={})[source]¶: Computes number of significant places

niimpy.preprocessing.location.location_significant_place_features(df, config={})[source]¶

Calculates features related to Significant Places.

Parameters

df: dataframe with date index
config: A dictionary of optional arguments
Optional arguments in config:: longitude_column: The name of the column with longitude data in a floating point format. Defaults to ‘double_longitude’. latitude_column: The name of the column with latitude data in a floating point format. Defaults to ‘double_latitude’. speed_column: The name of the column with speed data in a floating point format. Defaults to ‘double_speed’. resample_args: a dictionary of arguments for the Pandas resample function. For example to resample by hour, you would pass {“rule”: “1H”}.

niimpy.preprocessing.location.number_of_significant_places(lats, lons, times)[source]¶

Computes number of significant places.

Number of significant plcaes is computed by first clustering the locations in each month and then taking the median of the number of clusters in each month.

It is assumed that lats and lons are the coordinates of static points.

Parameters

latspd.DataFrame: Latitudes
lonspd.DataFrame: Longitudes
timesarray: Array of times
Returnsthe number of significant places discovered

niimpy.preprocessing.location.reset_groups(df)[source]¶: Group the dataframe by a standard set of columns listed in group_by_columns.

niimpy.preprocessing.sampledata module¶

Sample data of different types

niimpy.preprocessing.screen module¶

niimpy.preprocessing.screen.duration_util_screen(df)[source]¶

This function is a helper function for other screen preprocessing. The function computes the duration of an event, based on the classification function event_classification_screen.

Parameters

df: pandas.DataFrame: Input data frame

Returns

df: dataframe: Resulting dataframe

niimpy.preprocessing.screen.event_classification_screen(df, config)[source]¶

This function is a helper function for other screen preprocessing. The function classifies the screen events into four transition types: on, off, in use, and undefined, based on the screen events recorded. For example, if two consecutive events are 0 and 3, there has been a transition from off to unlocked, i.e. the phone has been unlocked and the events will be classified into the “use” transition.

Parameters

df: pandas.DataFrame: Input data frame
config: dict: Dictionary keys containing optional arguments for the computation of scrren information. Keys can be column names, other dictionaries, etc. The functions needs the column name where the data is stored; if none is given, the default name employed by Aware Framework will be used. To include information about the resampling window, please include the selected parameters from pandas.DataFrame.resample in a dictionary called resample_args.

Returns

df: dataframe: Resulting dataframe

niimpy.preprocessing.screen.extract_features_screen(df, bat, features=None)[source]¶

This function computes and organizes the selected features for screen events that have been recorded using Aware Framework. The function aggregates the features by user, by time window. If no time window is specified, it will automatically aggregate the features in 30 mins non-overlapping windows.

The complete list of features that can be calculated are: screen_off, screen_count, screen_duration, screen_duration_min, screen_duration_max, screen_duration_median, screen_duration_mean, screen_duration_std, and screen_first_unlock.

Parameters

df: pandas.DataFrame: Input data frame
features: dict: Dictionary keys contain the names of the features to compute. If none is given, all features will be computed.

Returns

computed_features: dataframe: Resulting dataframe

niimpy.preprocessing.screen.group_data(df)[source]¶: Group the dataframe by a standard set of columns listed in group_by_columns.

niimpy.preprocessing.screen.reset_groups(df)[source]¶: Group the dataframe by a standard set of columns listed in group_by_columns.

niimpy.preprocessing.screen.screen_count(df, bat, config=None)[source]¶

This function returns the number of times, within the specified timeframe, when the screen has turned off, turned on, and been in use. If there is no specified timeframe, the function sets a 30 min default time window. The function aggregates this number by user, by timewindow.

Parameters

df: pandas.DataFrame: Input data frame
bat: pandas.DataFrame: Dataframe with the battery information
config: dict: Dictionary keys containing optional arguments for the computation of scrren information. Keys can be column names, other dictionaries, etc. The functions needs the column name where the data is stored; if none is given, the default name employed by Aware Framework will be used. To include information about the resampling window, please include the selected parameters from pandas.DataFrame.resample in a dictionary called resample_args.

Returns

df: dataframe: Resulting dataframe

niimpy.preprocessing.screen.screen_duration(df, bat, config=None)[source]¶