update ressult
This commit is contained in:
161
data/task/cni/inference_out/README.md
Normal file
161
data/task/cni/inference_out/README.md
Normal file
@@ -0,0 +1,161 @@
|
||||
# Document Field Extraction Evaluation Results
|
||||
|
||||
## Overview
|
||||
|
||||
This document presents the evaluation results for document field extraction using different preprocessing approaches. The evaluation was conducted on a dataset of 56 document samples with various field types commonly found in identity documents.
|
||||
|
||||
## Evaluation Metrics
|
||||
|
||||
The evaluation uses standard information extraction metrics:
|
||||
- **Precision**: Ratio of correctly extracted fields to total extracted fields
|
||||
- **Recall**: Ratio of correctly extracted fields to total ground truth fields
|
||||
- **F1-Score**: Harmonic mean of precision and recall
|
||||
- **Accuracy**: Overall field-level accuracy
|
||||
- **TP**: True Positives (correctly extracted fields)
|
||||
- **FP**: False Positives (incorrectly extracted fields)
|
||||
- **FN**: False Negatives (missed fields)
|
||||
|
||||
## Preprocessing Approaches
|
||||
|
||||
### 1. No Preprocessing (Baseline)
|
||||
- **Configuration**: Raw images without any preprocessing
|
||||
- **Performance**:
|
||||
- Micro Precision: 79.0%
|
||||
- Micro Recall: 68.7%
|
||||
- Micro F1: 73.5%
|
||||
- Field Accuracy: 68.7%
|
||||
|
||||
### 2. Crop
|
||||
- **Configuration**: Content-aware cropping (no shadow removal)
|
||||
- **Performance**:
|
||||
- Micro Precision: 94.8%
|
||||
- Micro Recall: 89.9%
|
||||
- Micro F1: 92.3% (+18.8% improvement)
|
||||
- Field Accuracy: 89.9%
|
||||
|
||||
### 3. Crop + PaddleOCR + Shadow Removal
|
||||
- **Configuration**: Cropping with PaddleOCR document processing and shadow removal
|
||||
- **Performance**:
|
||||
- Micro Precision: 93.6%
|
||||
- Micro Recall: 89.4%
|
||||
- Micro F1: 91.5% (+18.0% improvement)
|
||||
- Field Accuracy: 89.4%
|
||||
|
||||
### 4. Crop + PaddleOCR + Shadow Removal + Cache
|
||||
- **Configuration**: Cropping with PaddleOCR, shadow removal, and caching
|
||||
- **Performance**:
|
||||
- Micro Precision: 92.5%
|
||||
- Micro Recall: 88.3%
|
||||
- Micro F1: 90.3% (+16.8% improvement)
|
||||
- Field Accuracy: 88.3%
|
||||
|
||||
### 5. Crop + Shadow Removal + Cache
|
||||
- **Configuration**: Cropping with shadow removal and caching
|
||||
- **Performance**:
|
||||
- Micro Precision: 93.6%
|
||||
- Micro Recall: 88.5%
|
||||
- Micro F1: 91.0% (+17.5% improvement)
|
||||
- Field Accuracy: 88.5%
|
||||
|
||||
## Field-Level Performance Analysis
|
||||
|
||||
### High-Performance Fields
|
||||
Fields that consistently perform well across all approaches:
|
||||
|
||||
| Field | Best F1 | Best Approach | Performance Trend |
|
||||
|-------|----------|---------------|-------------------|
|
||||
| **Gender** | 85.1% | Crop + PaddleOCR | Consistent improvement |
|
||||
| **Birth Date** | 80.5% | Crop + PaddleOCR | Strong improvement |
|
||||
| **Document Type** | 85.4% | Crop + PaddleOCR | Significant improvement |
|
||||
| **Surname** | 82.9% | Crop + PaddleOCR | Consistent improvement |
|
||||
|
||||
### Medium-Performance Fields
|
||||
Fields with moderate improvement:
|
||||
|
||||
| Field | Best F1 | Best Approach | Performance Trend |
|
||||
|-------|----------|---------------|-------------------|
|
||||
| **Birth Place** | 83.4% | Crop Only | Good improvement |
|
||||
| **Expiry Date** | 78.5% | Crop + PaddleOCR | Moderate improvement |
|
||||
| **Issue Date** | 69.3% | Crop + Shadow + Cache | Variable performance |
|
||||
| **Address** | 44.4% | Crop + PaddleOCR | Limited improvement |
|
||||
|
||||
### Low-Performance Fields
|
||||
Fields that remain challenging:
|
||||
|
||||
| Field | Best F1 | Best Approach | Notes |
|
||||
|-------|----------|---------------|-------|
|
||||
| **MRZ Lines** | 41.8% | Crop + Shadow + Cache | Complex OCR patterns |
|
||||
| **Personal Number** | 40.0% | Crop + PaddleOCR + Cache | Small text, variable format |
|
||||
| **Issue Place** | 50.0% | Crop + PaddleOCR + Cache | Handwritten text challenges |
|
||||
|
||||
### Zero-Performance Fields
|
||||
Fields that consistently fail across all approaches:
|
||||
- **Recto/Verso**: Document side detection
|
||||
- **Code**: Encoded information
|
||||
- **Height**: Physical measurements
|
||||
- **Type**: Document classification
|
||||
|
||||
## Key Findings
|
||||
|
||||
### 1. Preprocessing Impact
|
||||
- **Cropping alone** delivers the strongest overall boost (+18.8 F1 pts vs. baseline)
|
||||
- **PaddleOCR + Shadow Removal** is highly competitive (up to +18.0 F1 pts)
|
||||
- **Caching** has minimal to moderate impact on accuracy
|
||||
|
||||
### 2. Field Type Sensitivity
|
||||
- **Structured fields** (dates, numbers) benefit most from preprocessing
|
||||
- **Text fields** (names, addresses) show moderate improvement
|
||||
- **Complex fields** (MRZ, codes) remain challenging
|
||||
|
||||
### 3. Processing Pipeline Efficiency
|
||||
- **Crop** currently provides the best overall F1 in this evaluation
|
||||
- **Crop + PaddleOCR + Shadow Removal** is close and benefits some fields
|
||||
- **Caching** shows minimal gains; use for speed, not accuracy
|
||||
|
||||
## Recommendations
|
||||
|
||||
### For Production Use
|
||||
1. **Use Crop** as the primary preprocessing step
|
||||
2. **Focus optimization** on high-value fields (dates, document types, names)
|
||||
3. **Consider field-specific** preprocessing strategies for challenging fields
|
||||
|
||||
### For Further Research
|
||||
1. **Investigate MRZ line** extraction techniques
|
||||
2. **Explore advanced OCR** methods for handwritten text
|
||||
3. **Develop specialized** preprocessing for low-performance fields
|
||||
|
||||
### Performance Targets
|
||||
- **Overall F1**: Target 65%+ (currently 60.7% best)
|
||||
- **Field Accuracy**: Target 50%+ (currently 43.5% best)
|
||||
- **Critical Fields**: Ensure 80%+ F1 for dates and document types
|
||||
|
||||
## Technical Details
|
||||
|
||||
### Dataset Characteristics
|
||||
- **Total Samples**: 56 documents
|
||||
- **Field Types**: 25+ different field categories
|
||||
- **Document Types**: Identity documents, permits, certificates
|
||||
- **Image Quality**: Variable (scanned, photographed, digital)
|
||||
|
||||
### Evaluation Methodology
|
||||
- **Ground Truth**: Manually annotated field boundaries and text
|
||||
- **Evaluation**: Field-level precision, recall, and F1 calculation
|
||||
- **Aggregation**: Micro-averaging across all fields and samples
|
||||
|
||||
### Preprocessing Pipeline
|
||||
1. **Image Input**: Raw document images
|
||||
2. **Cropping**: Content area detection and extraction
|
||||
3. **Document Processing**: PaddleOCR unwarping and orientation
|
||||
4. **Shadow Removal**: Optional DocShadow processing
|
||||
5. **Field Extraction**: OCR-based text extraction
|
||||
6. **Post-processing**: Field validation and formatting
|
||||
|
||||
## Conclusion
|
||||
|
||||
The evaluation demonstrates that preprocessing significantly improves document field extraction performance. The **Crop + PaddleOCR** approach provides the best balance of performance and complexity, achieving a 14.1% improvement in F1-score over the baseline. While some fields remain challenging, the overall pipeline shows strong potential for production deployment with further field-specific optimizations.
|
||||
|
||||
---
|
||||
|
||||
*Last Updated: August 2024*
|
||||
*Evaluation Dataset: 56 document samples*
|
||||
*Total Fields Evaluated: 900+ field instances*
|
||||
Reference in New Issue
Block a user