KI4Tools – Artificial Intelligence for Hand Tools

The intelligent tools are underpinned by core technology consisting of a specially designed module with sensors to measure acceleration, rotation rate, and magnetic fields as well as a microphone, whose use is optional. This retrofittable solution is both compact and economical. It can be attached easily to a wide range of hand-operated tools, thereby paving the way for flexible working and universal applications. The module uses Wi-Fi or Bluetooth to transmit a steady stream of data, which form the basis for an AI pipeline that detects and identifies all operations.

Here, an initial binary classifier works as an action detector and starts by filtering out all time intervals that contain actions relating to the corresponding tool (such as the tightening or loosening of a screw using the electric screwdriver). The output provided to the user then consists of “action windows,” which contain the sensor data for the identified time intervals. A second, AI-based classifier subsequently receives these action windows and identifies the specific action to which the filtered data segment pertains. The position of the tool in the room at the time when the action takes place is also pinpointed.

An additional positioning system (QR code tracking), which is independent of the sensor module, also ensures that operations are carried out at the correct location and/or in the right order. An app provides the operator with real-time feedback on the progress of their work and informs them immediately if they deviate from the stipulated process. Another noteworthy feature is that these processing chains are trained using automatic machine learning, so that the entire process of training and evaluating new models is fully automated and requires no expert knowledge. In the future, the aim is for processes to be optimized (by continual learning) at the touch of a button via the existing app and for measurement data to be annotated semi-automatically. This is with a view to making the applied models more robust and more powerful on an ongoing basis.