After identifying image pixels as being part of the threadstructure, it is necessary to assemble them into an actualthread pattern. This is achieved by performing a registrationof the model prior, describing the desired thread pattern, tothe actually detected thread pattern. It is essential for thisstep to be robust against potential outliers and/or missed stitchpositions.Since the observed thread pattern will not equal the modelprior in general due to possible tissue distortions, an adaptationof the model is necessary.
The adaptation process correspondsto finding a unique deformation vector for each model stitchpoint. The corresponding processing pipeline is visualized inFigure 4.A. Thread RepresentationPositions of the thread appearing in the image are extractedusing a blob detection on the binary imagewges(x,y). Everyposition found this way will in the following be called a threadrepresentative. The representatives are preferably distributed inequidistant steps over the length of the thread. Furthermore,they can be ordered to form a sequence of points, representingthe entirety of the thread. The result is displayed in Figure 5c,with the circles visualizing the found representatives. However,it can be seen that outliers are possible (visualized in red).Additionally, individual thread positions might be missed.The model fitting and adaptation is not performed using eachindividual stitch, since such an approach would be highlysusceptible to interferences like outliers. Instead, more abstractfeatures are considered, that can be robustly recognized withinboth the set of representatives and the pattern model.•Characteristic pointsmay be thread endings, pointswhere the thread pattern changes abruptly its direction,or the intersection of lines.•Polygonsmay be formed by the linear connection ofneighboring representatives. The corresponding polygonswithin the pattern model are formed by the linear con-nection of neighboring model stitch positions.Next to the improved robustness, the computation complexityis heavily reduced.B. Initial Model-Based RegistrationThe specimen placed inside the inspection system may bearbitrarily rotated and shifted. For this purpose, an initialestimation of the rough placement of the thread patternrelative to the prior model is estimated. The estimationutilizes the generalized Hough transformation (GHT)  ,which has been proven to fulfill this task. The goal isto compute a global shift vector minimizing the distancebetween model and real world thread features.C. Iterative registrationBased on the initial model-based registration, the modeladaptation is performed in an iterative procedure. Each itera-tion consists of multiple steps. First, an assignment betweenfeatures within the model and the representatives is estab-lished. The assignment needs to be considered separately forboth feature types. The target of a characteristic model pointis a characteristic thread representative having the same typeand being at the minimum Euclidean distance. The search forcorresponding target polygons is based on the polygon centerand its direction.Once the assignments are established, a transformation ofthe current model features is performed to approximate thethread pattern. Within the first iterations, the assignmentsare rather unreliable. VS Enterprises Therefore, only a rigid registrationhaving few transformation parameters but many assignedfeature points is determined. Over the course of the iterations,the assignments become more reliable and the number offree transformation parameters is increased. In the end, theassignments become extremely reliable. Thus, individualmodel points are only then allowed to be shifted towardsindividual thread points, resulting in a controlled deformationof the model to adapt to the real thread pattern.Everytransformationisestimatedusinganenergyminimization approach, independent of the degree offreedom. There exist two types of energies, internal andexternal. Theinternal energyis a measure of the deformationof the model. The more a model polygon vector deviates fromits original vector in size and direction, the higher the cost.The contribution of theexternal energydepends on the typeof feature. If an assignment between a characteristic point inthe model and its counterpart on the thread is possible, themodel point can be directly attracted to its target. The strengthof attraction is independent of the direction and depends onlyon the Euclidean distance between both. For most cases, anexact assignment from model polygons to individual threadpolygons is not useful due to the high number of them thatcauses confusability. However, a direction of attraction canbe determined. Therefore the external energy for polygonsand the points spanning those polygons is computed independance on the direction of the model polygon normal at its center point and the projection of that normal onto athread polygon vector.1) Rigid registration:A rigid registration consists of a rota-tion and translation. Scaling is determined and applied as partof the deformation registration later on. The transformationis the result of an optimization over all model features andtheir associated targets, such that a non-linear equation systemneeds to be solved. An optimal solution is found using theconjugate gradient method .2) Deformable registration:New positions for the modelpoints are the result of an optimization procedure that min-imizes the total energy depending on the new positions. Alinear equation system with the coordinates of all modelpoints needs to be optimized. The relative weighting betweenexternal and internal energy determines how near a modelpoint is attracted to its target with the model steepness asthe constraint. https://vssewingmachine.in/ A large weight for the internal energy means avery steep model that is robust but fails to model each localdetail of the thread deformation. A small weight means a veryflexible but not very robust model. Hence, the improvementof the quality of the found targets with each iteration allowsa dynamic adaptation of the weights, resulting in a modeladapting better and better to local deformations recorded inthe thread image.The finally obtained deformation vectors between thread pat-tern and model prior are shown in Figure 5d.
Given arbitrary complex thread patterns, the system auto-matically detects the real world thread pattern and compares itwith the pattern intended by the manufacturer. A deformationvector can be calculated for every individual stitch with anaccuracy of up to50μm.The system may serve as a stand-alone for quality assurance.However, it also lays the foundation for the automated cor-rection of stitch positions in textiles distorted by the elasticityof the material. Based on the computed deformation vectorfield, an automated correction of the CNC program to createthe desired pattern is possible.An obvious limitation of the proposed system is the require-ment for the thread to appear visually different than thebackground tissue. However, this is not the case for a variety ofproducts, for which the thread color is wanted to be identicalto the background color. A typical example is a black carseat sewed using a black thread. We are currently workingon exchanging the RGB-camera for a multi-spectral imageacquisition in order to distinguish originally metamer, i.e.identical appearing, objects.
A camera based inspection system was introduced to auto-mate the quality inspection process within the area of CNCsewing. Created thread patterns are automatically detected andcompared against the intended pattern model. A deformationvector is computed for every individual stitch position. Forboth the thread detection as well as the model-based regis-tration and adaptation, dedicated image processing pipelineswere proposed.