Build User-Centered Systems to Protect Data Privacy!
My research lies at the intersection of human-computer interaction, privacy, and mobile systems, focusing primarily on two contrasting but complementary questions:
- Privacy: How can we make it easier for users, developers, and auditors to protect data privacy?
- Sensing: How can we build sensor-based/data-driven systems to improve our lives?
My research moves the field forward using a dual approach, inspired by the ancient Yin-Yang principle. When designing a new sensing system, I treat privacy as a first-class citizen. I often start with the best privacy practice and develop novel solutions to overcome other challenges. Meanwhile, my sensing research helps me understand the strengths and weaknesses of various privacy-intrusive technologies, allowing me to design privacy protections that balance the interests of users, developers, and auditors.
Bio.
I am a final-year Ph.D. student in the Human-Computer Interaction Institute at Carnegie Mellon University, advised by Jason Hong and Swarun Kumar. I also collaborate closely with Yuvraj Agarwal over the years. Before coming to CMU, I had been a research engineer at Yahoo Research for 3 years. At Yahoo, I developed key algorithms in visual search, cryptography, speech recognition and large-scale data mining. I also have run a startup and interned at Microsoft Research Asia, Bosch Research and Chinese Academy of Science before I started my Ph.D..
I received the UbiComp Gaetano Borriello Outstanding Student Award in 2020, which is given to one graduate student per year who has made outstanding research andservice contributions to the field of ubiquitous computing. My research has also been recognized by Research Highlights at Communications of the ACM and GetMobile, and best paper awards at Ubicomp and ACM Computing Reviews.
Featured projects
across human-computer interaction, privacy, and mobile systems.
Lean Privacy Review
A fast and cheap method to conduct “Front Page Test” systematically for Data Ethics.
Modular Privacy Flows
(Thesis research)
A software pattern that can minimize data collection and offer transparency and control.
- [Paper under review]
Software-Defined Cooking
A closed-loop system that can sense and control heating at a fine-grained resolution.
Full Conference and Journal-first Papers
Peekaboo: A Hub-Based Approach to Enable Transparency in Data Processing within Smart Homes
Haojian Jin, Gram Liu, David Hwang, Swarun Kumar, Yuvraj Agarwal, Jason I. Hong (Oakland’22).
Haojian Jin, Boyuan Guo, Rituparna Roychoudhury, Yaxing Yao, Swarun Kumar, Yuvraj Agarwal, Jason Hong (CHI’22).
Lean Privacy Review: Collecting Users’ Privacy Concerns of Data Practices at a Low Cost
Haojian Jin, Hong Shen, Mayank Jain, Swarun Kumar, Jason Hong (TOCHI’21).
Software Defined Cooking using a Microwave Oven
Haojian Jin, Jingxian Wang, Swarun Kumar, and Jason Hong. (MobiCom ’19).
[paper] [presentation] [demo paper] [video]
MobiPurpose: Inferring the Purposes of Network Traffic in Mobile Apps
Haojian Jin, Minyi Liu, Kevan Dohia, Yuanchun Li, Gaurav Kumar Srivastava, Matthew Fredrikson, Yuvraj Agarwal and Jason Hong. (IMWUT ’18 / UbiComp ’19).
[paper] [presentation] [data set] [traffic harness tools] [website]
WiSh: Towards a Wireless Shape-aware World using Passive RFIDs
Haojian Jin*, Jingxian Wang*, Zhijian Yang, Swarun Kumar, and Jason Hong (MobiSys ’18).
*co-primary authors. [paper] [presentation] [talk video] [project video]
MobiPurpose: Inferring the Purposes of Network Traffic in Mobile Apps
Haojian Jin, Minyi Liu, Kevan Dohia, Yuanchun Li, Gaurav Kumar Srivastava, Matthew Fredrikson, Yuvraj Agarwal and Jason Hong. (IMWUT ’18 / UbiComp ’19).
[paper] [presentation] [data set] [traffic harness tools] [website]
ElasticPlay: Interactive Video Summarization with Dynamic Time Budgets.
Haojian Jin, Yale Song, and Koji Yatani (ACMMM ’17).
[paper] [video] [presentation] [live demo]
Corona: Positioning Adjacent Device with Asymmetric Bluetooth Low Energy RSSI Distributions.
Haojian Jin, Cheng Xu, and Kent Lyons (UIST ’15).
[paper] [presentation] [video] [open source by ota42y]
Tracko: Ad-hoc Mobile 3D Tracking Using BLE and Inaudible Signals for Cross-Device Interaction.
Haojian Jin, Christian Holz, Kasper Hornbæk (UIST ’15).
ReviewCollage: A Mobile Interface for Direct Comparison Using Online Reviews.
Haojian Jin, Tetsuya Sakai, and Koji Yatani (MobileHCI 2014).
[paper] [presentation] [video]
Designing Alternative Representations of Confusion Matrices to Support Non-Expert Public Understanding of Algorithmic Outputs and Performance
Hong Shen, Haojian Jin, Ángel Alexander Cabrera, Adam Perer, Nina Balcan, Jason Hong. (CSCW ’20).
[paper]
‘I Can’t Even Buy Apples If I Don’t Use Mobile Pay?’ When Mobile Payments Become Infrastructural in China
Hong Shen, Cori Faklaris, Haojian Jin, Laura Dabbish, Jason Hong. (CSCW ’20).
[paper]
RFID Tattoo: A Wireless Platform for Speech Recognition
Jingxian Wang, Chengfeng Pan, Haojian Jin, Vaibhav Singh, Jason Hong, Carmel Majidi, Swarun Kumar. (IMWUT ’19/UbiComp’ 20).
[paper]
Sozu: Self-Powered Radio Tags for Building-Scale Activity Sensing
Yang Zhang, Yasha Iravantchi, Haojian Jin, Swarun Kumar, Chris Harrison. (UIST ’19).
[paper]
Finding Weather Photos: Community-Supervised Methods for Editorial Curation of Online Sources
David A. Shamma, Lyndon Kennedy, Jia Li, Bart Thomee, Haojian Jin, Jeff Yuan (CSCW’16).
[paper]
The Cohort and Speechify Libraries for Rapid Construction of Speech Enabled Applications
Tejaswi Kasturi, Haojian Jin, Aasish Pappu, Sungjin Lee, Beverly Harrison, Ramana Murthy, Amanda Stent (SIGDIAL 2015)
Mining Touch Interaction Data in Mobile Search
Qi Guo, Haojian Jin, Dmitry Lagun, Shuai Yuan, and Eugene Agichtein (SIGIR 2013).
Pushing the Range Limits of Commercial Passive RFIDs
Jingxian Wang, Junbo Zhang, Rajarshi Saha, Haojian Jin, Swarun Kumar. (NSDI ’19).
[paper]
Full Conference and Journal-first Papers
Sensor as a Company: On Self-Sustaining IoT Commons.
Haojian Jin, Swarun Kumar, Jason Hong. Technical Report.
[paper]
Providing Architectural Support for Building Privacy-Sensitive Smart Home Applications.
Haojian Jin, Swarun Kumar, Jason Hong. (UbiComp 2020 Doctoral Colloquium).
[paper]
SDC: Software-define Cooking using a Microwave oven.
Haojian Jin, Jingxian Wang, Swarun Kumar, Jason Hong. MobiCom 2019 Demo.
[paper]
RFWear: Towards Wearable Everyday Body-Frame Tracking using Passive RFIDs.
Haojian Jin, Jingxian Wang, Zhijian Yang, Swarun Kumar, Jason Hong. (Ubicomp 2018 Demo).
[paper]
Enhancing Email Functionality using Late Bound Content.
Haojian Jin, Vita Chen, Ritwik Rajendra, Jason Hong. Technical Report.
Advances and Challenges in Ad-hoc Mobile Tracking for Seamless Interaction across Commodity Devices.
Haojian Jin, Christian Holz. (CHI 2016 Workshop: Cross-Surface 2016)
Towards Estimating Web Search Result Relevance from Touch Interactions on Mobile Devices.
Qi Guo, Haojian Jin, Dmitry Lagun, Shuai Yuan, Eugene Agichtein. (CHI EA 2013)
Patents
Haojian Jin, Zheng Wen, Yale Song. Content-adaptive digital content adjustment method and system.
Granted in Apr. 2018, U.S. Patent number 9,942,581.
Christian Holz, Haojian Jin. System and method for 3D tracking for ad-hoc cross-device interaction.
Granted in Mar. 2018, US Patent number 9,929,817.
Haojian Jin, Christian Holz. System and method for detection of indoor tracking units.
Granted in Nov. 2017, US Patent number 9,813,854.
Christian Holz, Marius Knaust, Rajiv Ayyangar, Senaka Buthpitiya, Haojian Jin. User authentication and data encryption.
Granted in Nov. 2017, US Patent 9,817,956.
Haojian Jin, Kent Lyons, Cheng Xu. System and method for device positioning with bluetooth low energy distributions.
Granted in Mar. 2017, U.S. Patent number 9,602,956.
Jia Li, Haojian Jin. Mobile device image acquisition using objects of interest recognition.
Granted in January 2017, US Patent number 9,554,030.
Haojian Jin, Christian Holz. System and method for calibrating bluetooth low energy signal strengths.
Granted in July, 2016, US Patent number 9,385,821.
Haojian Jin, Zhijian Yang, Swarun Kumar, Jason Hong. System and Method for Tracking a Body.
Published in 2019.
Haojian Jin, Jingxian Wang, Swarun Kumar, Jason Hong. System and method for tracking a shape.
Published in 2019.
Haojian Jin, Jingxian Wang, Swarun Kumar, Jason Hong. System and Method for Heating an Item in a Microwave Oven.
Published in 2020.
Fun Projects
Pattern Colorizations for secret garden
A 10-line python code hack can auto-generate a random pattern colorization for the famouse coloring book – “Secret Garden“. I wrote the script to understand if shape is more important that colorization.
How many nodes required to acheive a 95% packing density if overlapping is allowed?
A traditional circle packing is an arrangement of circles inside a given boundary such that no two overlap and some (or all) of them are mutually tangent. This interesting problem was posted on Zhihu. I found it very interesting and pratical when placing the wireless nodes in a limited space. The picture left is the one with 95% packing density.
Toy robot car
The robots on the market are not extensible enough for further development. I built that robot based on the “plug in” software engineering concept and designed it to automate the radio signal collection.
[github]
CooTrading: Imagine a world where technology enables a sustainable environment.
CooTrading is a low-cost, energy efficiency and flexible hardware/software environmental solution with many well designed features. This prototype system allowed users to have a better Energy Consumption awareness and eventually put these users in charge of these consumptions.
4/61 in World Final (Paris, France), China Champion (1/500+), Microsoft Imagine Cup Software Design Competition
Realtime Gesture Recognizer
Different with most past work in Gesture Recognition, this toolkit focuses in predicting the user’s intention before they had finished their gesture. Similar with $1 gesture recognition, this gesture recognition also features in there is no requirement for library, toolkit, and training.
[github]
Research Reflections
Through my research, I want to define the boundaries within which possible futures must lie.
– I documented my reflection regarding a connected world here.
Privacy in a ubiquitously connected world.
It is clear that privacy will be a future requirement of all system software. Existing software architectures often require developers to build privacy provenance features individually, and users can only manage their privacy through the functions implemented by the software developers. However, these individually built features often take a lot of effort to build and result in a great deal of interface inconsistency, if the developers even considered privacy at all.
My thesis focuses on providing architectural support for building privacy-sensitive ubicomp apps. In the proposed architecture, developers can expend less effort on building privacy features, and users can have unified and consistent control/awareness of their personal data, which is independent of the features offered by the developers.
Being connected is not enough.
Being connected barely makes the physical world a remote-controllable platform. The connected world’s primary value would be on the numerous automation and optimizations enabled by the digital representation.
Towards this problem, I am building (1) the digital representation for real-world tasks, (2) the automation that can save people from tedious and repetitive tasks, and (3) the optimization that can do the jobs that most people cannot do.
Towards trillions of connected device in the world.
In the foreseeable future, the number of connected devices will continue to increase as long as (1) the value of each device can justify the cost; (2) users can manage these devices; (3) and the technical architecture can support the communication. But, supporting trillions of devices will require a fundamentally different architecture, which requires us to further push all three perspectives.
I am working on a set of projects to enable new applications based on sensor collaboration, allow individual users to manage thousands of devices, and design modern architecture to scale the device operations.
Each device in the future should be associated with a position.
Positioning is more than navigation, but a critical proxy to understand the context in ubiquitous computing. Allowing each device to know the nearby devices/people’s fine-grained positions can unlock numerous application possibilities, such as cross-device interaction, user activity recognition, etc.
I argue that we should architect a ubiquitous spatial tracking layer into the future operating system for each connected device, providing spatial awareness access like GPS API today. As the developer calls the positioning API, it will return a list of 2/3D positions of nearby devices/people with confidence scores.