What is OpenPack dataset?
OpenPacking Dataset is a new large-scale multi-modal dataset of packing processes. OpenPack is an open access logistics-dataset for human activity recognition, which contains 53 hours of human movement data with 20,129 instances of work operation annotation. In addition, OpenPack dataset contains data from IoT-enabled devices and rich metadata such as error annotation and order information.
The latest release is OpenPack Dataset v1.0.0. (Release Note)
Characterisitcs
Large Scale activity dataset in Industrial Domain
OpenPack is the first large-scale dataset for packaging work recognition. 16 distinct subjects packed 3956 items in 2048 shipping boxes in total and the total recording length is more than 53 hours. The work operations and actions were annotated by the professional annotators. The total number of annotated work operations and actions are 20,129 and 52,529 respectively.
+ IoT
Rich Modalities - Vision + Wearable + IoT Device
OpenPack provides 9 data modalities including acceleration, gyroscope, quaternion, blood volume pulse (BVP), and electrodermal activity (EDA) data, as well as keypoints, a LiDAR point cloud, and depth images. As for the wearable sensor modality, we collected data with 4 IMUs and 2 E4 sensors. For the vision modality, 2 depth cameras (front view and top view) and 1 LiDAR sensor were installed to our environment. In addition, we collected data from IoT devices which are sparse but have strong relation with specific activity classes.
Rich Meta Data
OpenPack also provides a rich set of metadata such as subject’s experience in packaging work and physical traits of subjects, enabling designs of various research tasks such as assessment of worker’s performance in addition to basic work activity recognition. In addition, we made a list of irregular activities which can be used for developing normal detection technologies.
Sample Video
Documentations (Index)
Here are links to the dataset documentations.
Acitivty Classes
Work Operations (10 classes)
| Image | ID | Operation |
|---|---|---|
| 100 | Picking | |
| 200 | Relocate Item Label | |
| 300 | Assemble Box | |
| 400 | Insert Items | |
| 500 | Close Box | |
| 600 | Attach Box Label | |
| 700 | Scan Label | |
| 800 | Attach Shipping Label | |
| 900 | Put on Back Table | |
| 1000 | Fill out Order | |
| 8100 | Null |
Data Collection Environments & Sensors
Download
OpenPack dataset is available in 3 repositories. Different repositories provide different sets of data with different licenses. Please go to openpack-dataset @GitHub and follow the instructions.
Sample Data
You can see samples of CSVs and images without downloading them. Sample data (U0209 - Session 5) is available on GitHub (openpack-dataset/data/openpack).
Preprocessed Dataset
Sensors used in the OpenPack dataset have different sampling rates. Therefore, each sensor is stored in a separate file. When you use them, you have to combine them using timestamps assosiated with each record. But we understand that it's not easy for the new users.
Therefore, we prepared pre-processed dataset for the quick trial. IMU data from 4 sensors and work operation labels are combined into one CSV file. This preprocessd dataset is available on zenodo (preprocessed-IMU-with-operation-labels.zip) (Click to Download [515MB]).
For more details, see data-stream/preprocessed.
Full Dataset
See DOWNLOAD.md (@GitHub:openpack-dataset).
Download Commands
Below is a command to download data from zenodo.
Terms of Use (Licence)
The OpenPack dataset consists of RGB data (OpenPack Dataset (+RGB)) and other data (OpenPack Dataset). The dataset without RGB data will continue to be provided under "CC BY-NC-SA 4.0". On the other hand, the dataset including RGB data will be provided under a different license for privacy and ethical considerations, as described below. Please check the license of the dataset you use before using it.
OpenPack Dataset
OpenPack Dataset by OpenPack Team is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License. This means that you must give appropriate credit, provide a link to the license, and indicate if changes were made. You may do so in any reasonable manner, but not in any way that suggests the licensor endorses you or your use. You may not use the material for commercial purposes.
OpenPack Dataset (+RGB)
“OpenPack Dataset (+RGB)” is one of the version that includes RGB data. OpenPack Dataset (+RGB) is licenced under “OpenPack Dataset (+RGB) License” . This means that you may not redistribute this data for any porpose. Also, the usage of this dataset is limited to the academic research only. Use of commercial purpose is prohibited. See the OPENPACK_DATASET_RGB_LICENSE.md (@GitHub:openpack-dataset) for the full license conditions.
Citation
When you use the OpenPack dataset or code please cite the following paper. (Updated on April 21 2024):
Naoya Yoshimura, Jaime Morales, Takuya Maekawa, Takahiro Hara, “OpenPack: A Large-scale Dataset for Recognizing Packaging Works in IoT-enabled Logistic Environments”. Proceedings of IEEE International Conference on Pervasive Computing and Communications (2024). (link)
Tutorials
Notebooks (Colab / Jupyter Notebook)
When you click on the link, a notebook will open in Google Colaboratory. A warning will appear when you run the first cell, but press "continue" to proceed. It is recommended to download the dataset to Google Drive when you run the tutorials on Google Colab. Before starting tutorials, please follow the instructions in the first tutorial (Tutorial - Download OpenPack Dataset to Google Drive) and download the OpenPack dataset from zenodo to your Google Drive.
This notebook shows how to train and test U-Net to predict work operations for each timeslot. You can learn the pipline of model training, e.g. loading data, training, predicting and visuaizing model outputs. This tutorial is prepared for the participants of the OpenPack Challenge 2022.
Paper
OpenPack: A Large-scale Dataset for Recognizing Packaging Works in IoT-enabled Logistic Environments
N. Yoshimura, J. Morales, T. Maekawa, T. Hara
Proceedings of the IEEE International Conference on Pervasive Computing and Communications (PerCom2024)
Unlike human daily activities, existing publicly available sensor datasets for work activity recognition in industrial domains are limited by difficulties in collecting realistic data as close collaboration with industrial sites is required. This also limits research on and development of methods for industrial applications. To address these challenges and contribute to research on machine recognition of work activities in industrial domains, in this study, we introduce a new large-scale dataset for packaging work recognition called OpenPack. OpenPack contains 53.8 hours of multimodal sensor data, including acceleration data, keypoints, depth images, and readings from IoT-enabled devices (e.g., handheld barcode scanners), collected from 16 distinct subjects with different levels of packaging work experience. We apply state-of-the-art human activity recognition techniques to the dataset and provide future directions of complex work activity recognition studies in the pervasive computing community based on the results. We believe that OpenPack will contribute to the sensor-based action/activity recognition community by providing challenging tasks. The OpenPack dataset is available at https://open-pack.github.io.
Preliminary investigation of SSL for Complex Work Activity Recognition in Industrial Domain via MoIL
Q. Xia, T. Maekawa, J. Morales, T. Hara, H. Oshima, M. Fukuda, Y. Namioka
Proceedings of the IEEE International Conference on Pervasive Computing and Communications Workshops and other Affiliated Events (PerCom Workshops 2024)
In this study, we investigate a new self-supervised learning (SSL) approach for complex work activity recognition using wearable sensors. Owing to the cost of labeled sensor data collection, SSL methods for human activity recognition (HAR) that effectively use unlabeled data for pretraining have attracted attention. However, applying prior SSL to complex work activities such as packaging works is challenging because the observed data vary considerably depending on situations such as the number of items to pack and the size of the items in the case of packaging works. In this study, we focus on sensor data corresponding to characteristic and necessary actions (sensor data motifs) in a specific activity such as a stretching packing tape action in an assembling a box activity, and try to train a neural network in self-supervised learning so that it identifies occurrences of the characteristic actions, i.e., Motif Identification Learning (MoIL). The feature extractor in the network is used in the downstream task, i.e., work activity recognition, enabling precise activity recognition containing characteristic actions with limited labeled training data. The MoIL approach was evaluated on real-world work activity data and it achieved state-of-the-art performance under limited training labels. (note: Best WIP Award @ PerCom2024)
Recent Trends in Sensor-based Activity Recognition
T. Maekawa, Q. Xia, R. Otsuka, N. Yoshimura, K. Tanigaki
Proceedings of the IEEE International Conference on Mobile Data Management (MDM)
This seminar introduces recent trends in sensor-based activity recognition technology. Technology to recognize human activities using sensors has been a hot topic in the field of mobile and ubiquitous computing for many years. Recent developments in deep learning and sensor technology have expanded the application of activity recognition to various domains such as industrial and natural science fields. However, because activity recognition in the new domains suffers from various real problems such as the lack of sufficient training data and complexity of target activities, new solutions have been proposed for the practical problems in applying activity recognition to real-world applications in the new domains. In this seminar, we introduce recent topics in activity recognition from the viewpoints of (1) recent trends in state-of-the-art machine learning methods for practical activity recognition, (2) recently focused domains for human activity recognition such as industrial and medical domains and their public datasets, and (3) applications of activity recognition to the natural science field, especially in animal behavior understanding.
Members
Special Thanks
Kana Yasuda
Annotator
Chikako Kawabe
Annotator
Makiko Otsuka
Annotator
Yashodmi Kaluarachchi
Annotator
Keisuke Tsukamoto
Web Developper
Pan Guangyang
Web Developper
Kohei Hirata
Web Developper
Yagi
Web Developper